Alignment of three-dimensional data collected in dental sessions, and applications thereof

ABSTRACT

Disclosed embodiments integrate a camera into an intraoral mirror. Integrating a camera into an intraoral mirror provides an efficient way to record and display what is visible to the healthcare provider in the mirror.

BACKGROUND Field

This field is generally related to dental instruments.

Related Art

Intraoral mirrors, also known as mouth mirrors, are among the mostfunctional and frequently used of dental instruments. Viewing objects ina mouth directly is difficult due to a limited, or perhaps nonexistent,line of sight. Intraoral mirrors allow a health care provider (HCP), forexample dentist, hygienist and others, to indirectly view teeth andother objects in a patient's mouth, such as the patient's gums andtongue, by observing their reflections in a mirror. Health careproviders use the intraoral mirror for a variety of tasks, including,but not limited to, evaluation and diagnosis, treatment selection, andeven to assist the treatment itself. A health care provider may useother tools, such as a dental hand piece, in conjunction with the mirrorto conduct procedures, such as tooth preparation, when the proceduresare conducted in areas that are not directly visible.

Not only are they used as a visual aid, intraoral mirrors are also usedas rigid tools to manipulate or protect objects in a patient's mouth.For example, a health care provider may use an intraoral mirror to shifta patient's cheek to make space for treatment or to expand the mouthspace for improved visibility. In addition, an intraoral mirror canprotect soft and hard tissue structures of a patient's mouth while otherparts of the mouth are treated.

Since an intraoral mirror is in contact with a patient's tissues insidetheir mouth, the mirror goes through sterilization after each treatment.In some cases, sterilization is done using a process known as“autoclaving.” Autoclaving subjects the mirror to high temperature andpressure, perhaps using steam. Because the mirror must be sterilizedafter each treatment, a dental office possesses multiple such mirrors.The mirror, made mostly of glass and metal, can withstand theautoclaving process. But, due to frequent use and sterilization, themirror eventually loses some of its clarity and its reflectiveness, thusneeding replacement.

In addition to intraoral mirrors, intraoral cameras are becoming morewidespread in dental clinics. Intraoral cameras have principally twouses. First, intraoral cameras are used to describe a diagnosis andexplain a possible treatment to a patient. For example, to explain adiagnosis or treatment, the health care provider may display images ofthe patient's mouth parts (e.g. teeth) to the patient. Second, theintraoral cameras are used to record the state of portions of thepatient's mouth. For example, a health care provider may capture aphotographic image of the patient's mouth before or after treatment.

Intraoral cameras are commonly shaped as pens, with an image capturedevice at their tip, pointed sideways. The tip helps orient the HCP asto where to position the camera to capture images of the desired area ofthe mouth. The captured images are presented in a display, and thenavigation is done using the display. However, because these displaysare the only viewfinder for the cameras, their use adds time to a dentalappointment. Additionally, in a common usage scenario, heath careproviders would commence a mouth inspection using a mouth mirror, if aneed to capture an image arises, the HCP would have to switch theintraoral mirror with an intraoral camera. This may seem a minor hassle,but in the busy environment of a dental clinic, it reduces the frequencyof capturing images.

Some dental procedures use dental composite resin material to gluefillings or build up structures on teeth during restoration procedures.After applying the material it is hardened using an instrument called alight cure. The light cure is used to illuminate the resin with lightwithin the spectrum of visible blue to ultraviolet. This light might beharmful to the eye, therefore an eye protector is used by the healthcareprovider while using the light cure. To perform such procedure, a healthcare provider applies the resin to the teeth. In many cases, to observethe tooth of interest, a health care provider uses a mouth mirror whileapplying the resin. When done, the health care provider switchesinstrument to a light cure, and illuminates the area for the resin tocure. When building up material on a tooth, the process repeats so thatresin is applied, followed by curing, and then applied again, requiringto repeatedly switch the mouth mirror and light cure instruments.

During a typical patient visit to a dental office, a health careprovider will record the patient's current dental status, also known asa dental tooth charting. A dental status, or dental tooth chart, is adiagram depicting the human teeth, where each tooth in the diagram ismarked to indicate an aspect of the tooth's condition. In examples, amarking may indicate that a tooth is missing, has had dental treatmentin the past, has a carious lesion, or has periodontal disease. Suchstatus is updated from time to time to reflect the patient's most up todate condition. By inspecting the diagram, a health care provider (HCP)may become quickly informed about a patient's dental health status.

Like other medical professions, dentists need to track their work byrecording and archiving with particularity a patient's dental conditionand treatments. To facilitate such recording and archiving, computerizedmedical record systems exist. Generally, before adding or modifying datain an archive, a health care provider logs into the system to identifyherself as the person providing the treatment. After logging in, thehealth care provider can select a patient record to retrieve informationon file or to add additional information to the patient's record, suchas a description of the current procedure, progress, and diagnoses forfuture treatment. The information in the patient's record is mostlytext. Images might also be recorded, but the accompanying text isessential for the record. Therefore, the input regarding the treatmentusually occurs at its end, when the professional is able to freely usethe computer keyboard, both for convenience and for proper hygiene. Someof these computerized medical record systems include a digital versionof a current status diagram, graphically depicting each tooth with avariety of marks and colors to represent various conditions.

Entering data into a patient's medical record takes time. However, inoperating a dental office, efficiency is important. Less time spent onadministrative tasks leaves more time for treatment. One of thehallmarks of a well-managed clinic is how efficiently time is used. As aresult, some dentists, despite having implemented a computerized systemas a management tool for their dental office, avoid the step of enteringa textual description of the session to the system, and keep handwrittennotes instead.

In dentistry (including orthodontics), multiple procedures exist thatrequire generating a three-dimensional (3D) model of a patient's mouth.At present, many of these models are created using physical material,such as the routine use of plaster models. Recently, technologies andproducts have emerged for the creation of a digital representation ofthe 3D modeling, the models commonly called “digital impressions.” Thesevarious products vary in the accuracy of the digital 3D model theyachieve and in the amount of time that a dentist (or other dental healthcare provider) has to spend during the dedicated scanning session.

The general structure of 3D scanning machines consists of a scanninghead, which performs intraoral measurements, from which depth measurescan be extracted (calculated), and connected to a computer unit whichprocesses the measurements to produce a digital model. The creation of a3D imprint of the mouth is necessarily performed in a dedicated session,during a time period allocated specifically for this task.

The computing resources, such as processor and memory capacity, neededto produce an imprint are substantial. The acquisition of depthmeasurements is done in sections, the size of each of the sections isdetermined according to the amount of concurrent depth information thatcan be acquired by the scanning head. Then, generally, a 3D model ofthat section is calculated. Later, a registration of the varioussections is performed, to allow the “stitching” of adjacent digitizedregions into one consistent model. Additionally, if colors are acquired,a process of coloring and of possibly adding texture is performed.Additional processes might also be employed. The computer processingpower required to produce such digital imprint in a reasonable amount oftime is, therefore, substantial.

The technology implemented in the scanner head affects both accuracy andspeed. The simplest of these, at least when being measured in terms ofavailability of electronic components, is a scanner head with an imagecapturing camera.

Several algorithms for extracting depth information from sets of imagesexist, involving identification of matching points or shading extractionor other concepts, followed by solving a so-called “triangulationproblem”.

In practice, achieving accurate depth measurements through imageanalysis in a short period of time is challenging since much of theinformation that is extracted from images tends to exhibit geometricnoise, due to either distortion in the image capturing process, oruneven reflection of the illumination of intraoral objects. One of thereasons for the presence of such uneven illumination is that some areasmay exhibit Lambertian reflection, while other areas exhibit specularreflection. The presence of saliva in an intraoral environment, is anadditional, and sometimes substantial hurdle to the accuracy of theprocess, since it may cause some areas to exhibit specular reflectioneven if otherwise they would not. Using a multitude of captured imagesof an intraoral object from various angles, will greatly assist thetriangulation accuracy, but unfortunately will extend the computationaleffort and hence the amount of time and processing power required foraccurate calculation.

Systems and methods are needed to provide a more efficient way to show apatient images from the patient's mouth, to record the patient's dentalstatus, and to create digital dental impressions.

BRIEF SUMMARY

Embodiments integrate a camera into an intraoral mirror. Integrating acamera into an intraoral mirror provides an efficient way to record anddisplay what is visible to the healthcare provider in the mirror. Insome embodiments, the images taken from the intraoral mirror are used tocapture dental status and generate a digital dental impression.

In an embodiment, a method aligns three-dimensional dental data from aplurality of dental sessions. In the method, a plurality of point cloudsare received. Each point cloud represents three-dimensional points in adental room that were detected in the interior of a patient's mouth, andeach point cloud is collected during a different dental session when thepatient's mouth is located in a different position in the dental room.The point clouds are aligned to represent a common volume element withinthe patient's mouth. Based on the aligned plurality of point clouds, athree-dimensional model the interior of a patient's mouth is generated.

Other system, method, and computer program product embodiments are alsodisclosed.

Further embodiments, features, and advantages of the invention, as wellas the structure and operation of the various embodiments, are describedin detail below with reference to accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated herein and form partof the specification, illustrate the present disclosure and, togetherwith the description, further serve to explain the principles of thedisclosure and to enable a person skilled in the relevant art to makeand use the disclosure.

FIG. 1 is a diagram illustrating components of an intraoral mirrorincluding an integrated camera, according to an embodiment.

FIG. 2 is a diagram illustrating a cross-section of a mirror portion ofthe intraoral mirror.

FIG. 3 is diagram illustrating an intraoral mirror being used to examinea patient's mouth from the perspective of a healthcare provider.

FIG. 4 is a diagram illustrating how light from the patient's mouth isreflected off the mirror's reflective surface and observed by ahealthcare provider.

FIGS. 5A-C are diagrams illustrating how an orientation of an intraoralmirror may change as a healthcare provider rotates it.

FIG. 6A illustrates how the portion of an image captured by a wide-anglecamera that is visible to a healthcare provider on the reflectivesurface of the intraoral mirror is determined.

FIG. 6B illustrates how the portion determined to be visible to thehealthcare provider is rotated to adjust for rotation of the intraoralmirror.

FIGS. 7A-C illustrate how dazzle is minimized.

FIG. 8A illustrates a room of a dentist office including a system fordisplaying what is apparent to the healthcare provider in an intraoralmirror.

FIG. 8B is an architecture diagram of the system in FIG. 8A.

FIG. 9 is a block diagram of the intraoral mirror including anintegrated camera.

FIG. 10 is a flowchart illustrating an example operation of the systemof FIG. 8A.

FIG. 11 is a block diagram further detailing components of the system ofFIG. 8A.

FIG. 12 is a flowchart illustrating a method to estimate the portion ofthe image captured by an image sensor in the intraoral mirror that isvisible to a healthcare provider.

FIGS. 13 and 14 illustrate methods for rotating the image to adjust forthe orientation of the intraoral mirror.

FIG. 15A-B illustrate alternative methods to retrieve data from anintraoral mirror device.

FIG. 16 is a diagram illustrating how images from a stream of images ina video from the intraoral mirror are selected for capture.

FIG. 17A is a diagram illustrating how a feature from the interior of apatient's mouth appears in different images taken from the intraoralmirror from different perspectives.

FIG. 17B is a diagram illustrating how the matched features are used toalign the different images to be stitched into a panorama indicating apatient's dental status.

FIG. 17C is a diagram illustrating stitching the images captured fromthe image sensor into a panorama indicating the patient's dental status.

FIGS. 18A-C are diagrams illustrating how to capture an immersivepanorama and render it with a sense of depth.

FIGS. 19A and 19B are diagrams illustrating how varying illuminationfrom the intraoral mirror can provide information on thethree-dimensional shape of a patient's teeth.

FIG. 20 is a diagram illustrating how a smart mirror device may be movedduring a procedure.

FIGS. 21A-B are flowcharts illustrating a method for incrementalgeneration of a dental status and impressions.

FIG. 22 illustrates a system with a server to stitch panoramasindicating a patient's dental status and to generate dental impressions.

FIG. 23 illustrates a method for varying illumination of the intraoralmirror.

FIG. 24 illustrates a method for automatically adjusting illuminationoriginating from the intraoral mirror.

FIG. 25 illustrates a method for monitoring usage of the intraoralmirror.

FIG. 26 is a diagram illustrating an alternative configuration of theintraoral mirror where an image sensor is placed on a non-autoclavableportion of the mirror.

FIG. 27 is a diagram illustrating an alternative configuration of theintraoral mirror where an image sensor is placed on a jointed appendageof a non-autoclavable portion of the mirror.

FIG. 28 is a diagram illustrating an alternative configuration of theintraoral mirror where the autoclavable portion detaches into twopieces.

FIG. 29 is a block diagram illustrating an alternative embodiment of thesmart mirror.

The drawing in which an element first appears is typically indicated bythe leftmost digit or digits in the corresponding reference number. Inthe drawings, like reference numbers may indicate identical orfunctionally similar elements.

DETAILED DESCRIPTION

The detailed description that follows is divided into ten sections.First, an intraoral mirror with an integrated camera is described withrespect to FIGS. 1 and 2. Second, how the intraoral mirror is used as aviewfinder for the camera is described with respect to FIGS. 3, 4, 5A-C,6A-B and FIGS. 7A-C. Third, systems and methods that utilize theintraoral mirror are described with respect to FIGS. 8-16. Fourth, howthe intraoral mirror is used to capture a patient's dental status isdescribed with respect to FIGS. 17A-C and 18A-C. Fifth, how theintraoral mirror is used to capture a patient's dental impression isdescribed with respect to FIGS. 19A-B. Sixth, how the status andimpressions are generated incrementally is described with respect toFIG. 20 and FIGS. 21A-B. Seventh, a system for capturing a patient'sstatus and generating a patient's dental impression is described withrespect FIG. 22. Eighth various methods for adjusting illumination ofthe intraoral mirror are described with respect to FIGS. 23 and 24.Ninth, a method for monitoring usage to conserve the intraoral mirror'spower is described with respect FIG. 25. Tenth and finally, alternativeembodiments of the intraoral mirror are described with respect to FIGS.26-29.

A. INTRAORAL MIRROR WITH INTEGRATED CAMERA

FIG. 1 is a diagram illustrating components of an intraoral mirrorincluding an integrated camera, referred to herein as a smart mirrordevice, according to an embodiment. In particular, FIG. 1 illustratesphysical components of a smart mirror device 100.

Smart mirror device 100 includes two separate, detachable physicalcomponents: an oral piece 102 and a hand piece 110. The hand and oralpieces 102 and 110 may be further detachable into sub-pieces. Oral piece102 may be the portion of smart mirror device 100 that can enter apatient's mouth. Oral piece 102 may include portions of the smart mirrordevice 100 that can be sterilized through being autoclaved. To beautoclaved, oral piece 102 can withstand being placed in an autoclavemachine, which is a pressure chamber used to carry out industrialprocesses requiring elevated temperature and pressure different fromambient air pressure. Many autoclaves are used to sterilize equipmentand supplies by subjecting them to high-pressure saturated steam at121-132° C. (249-270° F.) for around 15-20 minutes depending on the sizeof the load and the contents. Additionally, oral piece 102 may bereplaceable, for example if the reflective surface of viewfinder mirror103 becomes damaged or due to an electronics component failure, whichmay occur as result of repeated use and sterilization.

In contrast to oral piece 102, hand piece 110 is not adapted to be incontact with a patient's mouth. Because hand piece 110 is not in contactwith a patient's mouth, it may not need to be as sterile as oral piece102. Hand piece 110 may include portions of the smart mirror device 100that cannot withstand being autoclaved. For example, hand piece 110 mayinclude sensitive electronics and a power supply that could be damagedby the heat, pressure, and moisture in an autoclave machine. Also,components may be placed on hand piece 110 as opposed to oral piece 102to avoid the need to replace them as they become worn out throughrepeated use and sterilization.

Returning to oral piece 102, oral piece 102 includes a viewfinder mirror103, an appendage 104, and a connector 105. Viewfinder mirror 103 is amirror, hence has a reflective surface. Viewfinder mirror 103 may beround in shape. Viewfinder mirror 103's reflective surface includes asemi-reflective surface 101. Semi-reflective surface 101 appearsreflective from the outside of the smart mirror device 100. But, frombehind semi-reflective surface 101, surface 101 is transparent. In anembodiment, surface 101 is a transparent feature, small enough to enablea healthcare provider to have a clear view of the reflection whenobserving the reflective surface. Thus, behind semi-reflective surface101, an image sensor (not shown) may be concealed. In one embodiment,viewfinder mirror 103 is round and semi-reflective surface 101 islocated at the center of mirror. Viewfinder mirror image 103 providesvisual guidance to a user about the objects that may be included inimages captured by the sensor concealed behind surface 101. Hence, whensmart mirror 100 is being used by a health care provider in an intraoralenvironment, the image sensor concealed behind surface 101 may captureimages that include at least some of the objects whose reflection can beobserved in the viewfinder mirror 103.

In addition to semi-reflective surface 101, viewfinder mirror 103 mayalso include a plurality of light sources 107. The light sources 107 areaffixed around the perimeter of viewfinder mirror 103, and possiblyconcealed behind its reflective surface. Light source 107 may illuminatethe intraoral environment. In some embodiments, light source 107illuminates in the visible light spectrum, for example a light ofsimilar color hues of daylight, which enables a visual perception (aswell as capturing of images) of natural colors, or maybe a so-called“warm white” color which in some cases produces an illumination which ismore comfortable to the human eye. In some embodiments, light source 107illuminates in non-visible radiation, for example in a frequency withinthe infrared spectrum.

Appendage 104 is affixed to viewfinder mirror 103, protecting hand piece110 from contacting the intraoral environment while allowing forcomfortable intraoral use of viewfinder mirror 103. Appendage 104 has anelongated shape with viewfinder mirror 103 affixed to one end of theshape, extending viewfinder mirror 103 into the patient's mouth. In someembodiments, appendage 104 has a shape of a tube or cylinder, but othershapes, including those with non-rounded cross-sections, are possible.In some embodiments, at least part of appendage 104 is hollow, allowingspace for electronic components. In some embodiments, electroniccomponents are located elsewhere, the invention is not so limited.

While one end of appendage 104 is affixed to viewfinder mirror 103, theopposite end is affixed to a connector 105. Connector 105 is a physicaland electrical connector to connect oral piece 102 with hand piece 110.In some embodiments, connector 105 is internal to appendage 104. Forexample, hand piece 110 can slide or partially slide into a hollow partof appendage 104 to attach hand piece 110 with connector 105. Animpermeable lid (not shown) may be used to seal off connector 105 whenoral piece 102 is autoclaved. Appendage 104 may interlock with handpiece 110 to avoid separation of oral piece 102 and hand piece 110during use of smart mirror 100. To interlock with hand piece 110, thetwo pieces may, for example, screw into one another, or pressure fromhand piece 110 sliding into appendage 104 may hold the two piecestogether during use.

As mentioned above, hand piece 110 may include components that are notcapable of being autoclaved. Thus, oral piece 102 may be detached fromhand piece 110 so that it can be autoclaved. Hand piece 110 includes aconnector 113, a handle 112 and a button 114. Connector 113 is adaptedto couple with connector 105 of oral piece 102. The coupling may beelectrical. For example, connector 113 may supply power to oral piece102 through connector 105. In addition, hand piece 110 and oral piece102 may transmit and receive data from one another via connectors 113and 105.

Handle 112 enables a health care provider to grasp smart mirror device100. Like appendage 104, handle 112 may have an elongated shape, such asa cylinder, and may be hollow on the inside to conceal electroniccomponents. For example, handle 112 may conceal a power supply, aprocessor, and accelerometer or gyroscope. On handle 112 is button 114,which accepts input from the health care provider. In some embodiments,handle 112 is completely covered by appendage 104, so a health careprovider does not literally grasp it when smart mirror device 100 is inuse.

FIG. 2 is a diagram illustrating a cross-section of viewfinder mirror103 of the smart mirror device 100 shown in FIG. 1. Viewfinder mirror103 includes a mirror surface 206 that reflects light into thehealthcare provider's field of view. As mentioned above for FIG. 1,mirror surface 206 comprises partially reflective surface 101, which isconfigured such that one side appears reflective to a health careprovider and that another side is transparent to an image sensor 204.Also as mentioned above for FIG. 1, around a perimeter of mirror surface206 may be a plurality of light sources 107A-107B. Light sources 107 maybe illumination devices, such as light emitting diodes. Like imagesensor 204, light sources 107 may be situated behind a partiallyreflective surface (not shown).

Situated between image sensor 204 and partially reflective surface 101is a lens 202. Lens 202 may refract light to image sensor 204 from afield of view 208. Lens 202's field of view 208 has a viewing angle thatis wide enough to simultaneously capture both part of a face of a healthcare provider and a portion of a patient's mouth that is visible to thehealth care provider in most intraoral usage conditions.

In various embodiments, the viewfinder mirror 103 may have severallenses. The multiple lenses may move or adjust to allow focus atdifferent distances. There may be a mechanism for autofocus or amechanism to add or remove one or more lenses between the mirror surfaceand the sensor.

In additional embodiments, light from surface 101 may be split tomultiple structures of lenses and sensors. In addition, the light fromsurface 101 may be folded before it reaches an image sensor 204 using aprism. As will be discussed below, the image sensor 204 may be locatedin an appendage or hand piece.

B. USING AN INTRAORAL MIRROR AS A VIEWFINDER

FIG. 3 shows diagram 300 illustrating an intraoral mirror being used toexamine a patient's mouth from the perspective of a healthcare provider.In particular, diagram 300 includes smart mirror device 302 having amirror surface 304. In diagram 300, smart mirror device 302 is beingused to inspect teeth 306. In diagram 300, mirror surface 304 ispositioned to show the lingual surface of teeth 306 (the back of thetooth) from the healthcare provider's perspective. Notably, what isvisible from the healthcare provider's perspective may differ from whatis visible from the image sensor's perspective. This is illustrated, forexample, in FIG. 4.

FIG. 4 shows a diagram 400 illustrating how light from a patient's mouthis reflected off a mirror's reflective surface and observed by ahealthcare provider. Diagram 400 shows how smart mirror device 302'smirror surface 304 reflects light from the lingual surface of teeth 306to an eye of a healthcare provider 412. In particular, a plurality ofrays of light spanning from ray 414 to ray 416 travel from teeth 306 tomirror surface 304. Each of the rays, which we shall call incident rays,reach mirror surface 304 at an incidence angle, for example ray 422 atincidence angle 404. Then, according to the so-called “law ofreflection”, the rays are reflected from the mirror surface 304 so thateach reflected ray, the respective incident ray and the normal to thesurface at the incidence point (labeled incidence point 424) are all onthe same plane. Moreover, the angle of reflection is equal to the angleof incidence. The combination of reflected rays that happen to reflecttoward the healthcare provider 412 define what is being observed in themirror. For example, incident ray 422 reaches the mirror surface 304 atan incidence point on mirror surface 304 and at an incidence angle 404and its reflected ray 420 towards the healthcare provider 412 is at areflection angle 402 which is equal to the incidence angle 404. Thus,the lingual surface of teeth 306 is visible to healthcare provider 412,and health care provider 412 observes the perspective illustrated inFIG. 3.

As described above for FIG. 2, smart mirror device 302 includes an imagesensor, and the image sensor captures light refracted by a lens withfield of view 208. As illustrated in diagram 400, field of view 208 iswide enough to capture the lingual surface of teeth 306, which isvisible to healthcare provider 412 on mirror surface 304, even in caseswere healthcare provider 412 is focusing on incidence points off thecenter of the mirror. Thus a health care provider may use a smart mirrordevice 302 to capture image snapshots without interrupting treatment byswitching to a separate intraoral camera instrument. Nonetheless, thecaptured image must be processed so that the result fits what healthcareprovider 412 observes in the mirror surface 304. This processing isrequired because the orientation of an intraoral mirror may change as ahealthcare provider rotates it and also because possibly more is visibleto the image sensor in smart mirror device 302 than is visible tohealthcare provider 412, for example, field of view 208 is wide enoughto capture part of the face of health care provider 412, while healthcare provider 412 may not witness his reflection in mirror surface 304.Therefore, in order to improve the use of the intraoral mirror as aviewfinder, the portion of an image captured by the image sensor that isbeing observed by the healthcare provider 412 must be determined.

To determine which portion of field of view 208 is being observed byhealthcare provider 412, healthcare provider 412's face may be detectedin the image captured by image sensor. Then, following the law ofreflection, only backwards, a ray 420 extended from health care provider412 toward an incidence point of the mirror is determined. Ray 420 mayhave an angle 402. While diagram 400 only identifies the angle ofincidence and reflection for a ray 420 and a ray 420 on a plane forclarity and simplicity, a skilled artisan would recognize that thegeometry is three-dimensional, and the drawing describes the angles forthese rays on the plane of incidence, which is perpendicular to thereflective surface at point of incidence 424. Based on ray 420, a ray422 extended from the mirror surface toward an object appearing to thehealth care provider on the mirror surface is determined. Ray 422 isdetermined by having an angle 404 equal to angle 402. An incidence point424 that corresponds to healthcare provider's 412 center of gaze definesthe center of the portion of interest out of field of view 208. Thecenter of gaze may be determined by eye tracking methods. A reasonableapproximation is to assume that healthcare provider 412 is observing anobject at the center of the mirror.

There are various ways to conduct the calculation above. One way toapproximate the calculation is illustrated in FIG. 6A. FIG. 6A shows adiagram 600 illustrating image 620 captured from an image sensor in asmart mirror device. Image 620 includes teeth 612 and a healthcareprovider's face 610. A skilled artisan would recognize that, given theorientation of the smart mirror device in FIG. 3, the image actuallycaptured from the image sensor may be rotated from what is presented inimage 620.

A face detection algorithm may be applied to identify which portion ofthe image includes the healthcare provider's face 610. The algorithm mayfurther be specified to identify not only the healthcare provider'sface, but more specific to identify only the healthcare provider's eyes.In diagram 600, the algorithm detects a face in a region 602, which hasa center point 604.

An origin having coordinate point (0,0) is determined at the center ofthe image. This may be the point where light that is normal to thecenter of the mirror surface is captured. Center point 604 may bedetermined relative to the origin as an (X, Y) coordinate. An example,the X and Y values could be the pixel values relative to the origin. TheX and Y distance may be expressed in other ways as well.

Next, a point 606 in image 620 is determined as a point having theinverse coordinate (−X, −Y). Point 606 may be the center of the portionthat is determined to be visible by the healthcare provider. With point606 at the center, a region 608 is determined. Region 608 should bedetermined such that it includes a portion 614 of image 620, which isvisible on the mirror surface. The size of region 608 may be fixed inaccordance with the mirror size or the target display. Alternatively, itmay be adjusted according to a distance between an object being viewedand the mirror surface, or the amount of relevant information in theimage.

The orientation of objects as they appear in region 608 depends on theorientation of objects as they appear in image 620, the orientation ofobjects as they appear in image 620 depends on the orientation of theimage sensor, and the orientation of the image sensor depends on theorientation of the smart mirror device. A healthcare provider holds thesmart mirror device at an angle relative to the object being observed.The angle is subject to variation as it changes among observations ofdifferent objects and possibly also of the same object. A healthcareprovider looking at the reflection of an image may change the angle toaccommodate better viewing. To such an observer, a slight change inmirror position or angle may not significantly alter the view of anobject in the mouth.

Conversely, for an image capturing device at the center of said mirror,the image being captured may dramatically change following even smallchanges to the angle, causing changes to the size and location in theimage of the object being observed in the mirror, such as a tooth, thatmight be in close proximity to the intraoral mirror. The visualinformation captured in the image will change when the angle changes.

A round mirror reflecting an object may rotate around its center, and tothe observer there will be no apparent change in the reflection. Thehuman interpretation of the reflection has a natural orientation,generally upwards. Conversely, for an image capture device at the centerof said mirror, a change in rotation also changes the direction that theimage sensor considers “upwards.” Therefore, the image being capturedmight appear substantially different than the image observed in themirror, possibly hampering the use of said mirror as a viewfinder forimage capture.

To deal with these issues, the smarter device may include an orientationmeasuring device, such as a gyroscope, that collects data to determinean orientation of the dental instrument. Based on the informationcollected from the gyroscope, the image may be reoriented, asillustrated FIGS. 5A-C and FIG. 6B.

FIGS. 5A-C are diagrams illustrating how an orientation of an intraoralmirror may change as a healthcare provider rotates it. FIG. 5A shows adiagram 500 illustrating the geometry of the problem being addressed bythis invention. Diagram 500 illustrates a portion of smart mirror device100 described in FIG. 1 including viewfinder mirror 103 and appendage104, which includes or is connected to a handle that the healthcareprovider uses to grasp a mirror. Viewfinder mirror 103 is coupled to animage sensor 204.

In diagram 500, smart mirror device 100 may be placed vertically at aninitial position. In an embodiment, the initial position may be in acradle on a charging station, where the orientation of the device isknown. Because smart mirror device 100 is vertical, an axis 505 whichcoincides with a diameter running from the top of smart mirror device100 through the center of viewfinder mirror 103 and towards the handleis known to face straight up-and-down. From here, the orientationmeasuring device may be calibrated. Downward projection 507 may beindicated as down toward appendage 104. In this orientation, the imagesensor 204 may be oriented such that when it captures an image of anobject 520, object 520 appears at the bottom of the captured image.

In diagram 500, a normal 506 is perpendicular to viewfinder mirror 103at its center, which, according to an embodiment, is where image sensor204 is located. The normal 506 points to the direction that the imagecapture sensor 204 is pointing, so that when capturing an image, anobject located in the direction of normal 506 will appear at the centerof the image.

FIG. 5B and FIG. 5C show diagrams depicting how smart mirror device 100in FIG. 5A can change orientation with respect to an object 520.

FIG. 5B shows how smart mirror device 100 is rotated around normal 506.As shown in FIG. 5B, object 520 appears at a different location withrespect to image sensor 204. Without correction, object 520 may appearin an awkward location in the resulting image captured from image sensor204.

To correct for the location of object 520, any images collected fromimage sensor 204 may be rotated by an angle 540 between line 507 andaxis 505 to ensure that line 507 continues to project downward. Angle540 may be the azimuth angle around normal 506 between axis 505 anddownward projection 507.

As mentioned above, angle 540 may be sensed by an orientation measuringdevice in smart mirror device 100. As mentioned above, the orientationmeasuring device may be a gyroscope able measure changes in yaw, pitch,and roll of smart mirror device 100. By integrating those changes,orientation measuring device can determine the yaw, pitch, and rollangles of smart mirror device 100. Depending on the position of thegyroscope within smart mirror device 100, the roll angle may be set to,or at least used to determine, angle 540 between axis 505 and downwardprojection 507.

FIG. 5C is a diagram 560 showing smart mirror device 100 being furtherrotated. As smart mirror device 100 is further rotated, the orientationmeasuring device senses that smart mirror device 100 has been rolledfurther and determines that the angle between axis 505 and downwardprojection 507 is greater, illustrated in diagram 560 as azimuth angle570. Based on this azimuth angle, the image captured from image sensor204 is rotated. In particular, the image captured from image sensor 204(or at least a portion thereof) is rotated by the roll calculated basedon measurements by the orientation measuring device—angle 540 in FIG. 5Bor angle 570 in FIG. 5C. This is illustrated, for example, in FIG. 6B.

FIG. 6B shows a diagram 650 illustrating how the region identified asbeing visible to a healthcare provider in a mirror is rotated. Similarto diagram 600 in FIG. 6A, diagram 650 shows an image 620 taken from theperspective of a smart mirror image sensor. Image 620 again includeshealthcare provider 610 and teeth 612. Once again, in diagram 600, aregion 608 is determined corresponding to what is visible to healthcareprovider 610 on the mirror surface. In this illustration also shown theregion as is rotated according to the way health care provider 610 holdsthe smart mirror device. Once region 608 is determined, it is rotatedaround its center by the azimuth angle determined by the smart mirror'sorientation measuring device to produce what is illustrated in diagram650 as region 652. Region 652 may be cropped from the image andidentified as the image visible to healthcare provider 610 on the smartmirror's surface.

In this way, by detecting a face in an image captured by a smart mirrordevice's image sensor and by detecting the smart mirror device'sorientation, a portion of the image captured by the smart mirrordevice's image sensor that corresponds to what is visible to thehealthcare provider on the mirror surface can be determined, thusdetermining the portion visible on the mirror surface from a largerfield of view.

Back to FIG. 3, a smart mirror device 302 also illuminates the patient'smouth. Such illumination improves the view of the mouth to a healthcareprovider and also enables enough light to arrive to the image sensor forcapturing clearer images. However, when gazing at the mirror whileinspecting a patient's mouth, some of the light sources 107 might bevisible to the healthcare provider, possibly disturbing his vision bydazzling (i.e., glare). Therefore, to improve the use of the intraoralmirror as a viewfinder, which lights sources out of the light sources107 of a smart mirror 302 should be lit and which should not be lit mustbe determined.

To determine which of light sources 107 should be lit and which shouldnot be lit, healthcare provider's face may be detected in the imagecaptured by the image sensor. Referring to FIG. 6A, in image 620 thedistance between center point 604 and the center of the image at (0,0)represents the angle between a normal to the mirror at its center andthe direction towards the healthcare provider's face. Accordingly, lightsources 107 that may emit light in that direction may be turned off toavoid dazzling. The angle of the direction may be spherical.

Another method to determine which of light sources 107 should be lit andwhich should not be lit is to find out the parts, or segments, of image620 that contain intra-oral objects. One way to do this is to comparetwo images taken at the same position but with different illuminations,for example, one with all of light sources 107 turned on and the secondwith all of light sources 107 turned off. Due to the so-called “inversesquare law”, stating that the intensity of illumination is inverselyproportional to the square of the distance from the light source, theeffect of turning on the illumination on the closer intra-oral objectswill be substantial, while the effect on farther, extra-oral objectswill be much smaller if noticeable at all. Therefore the image may besegmented into intra and extra-oral sections by analyzing thisdifference. Similarly, even when some of light sources 107 are turnedon, the use or addition of a colored light could be used, for examplealternating the illumination of a green or blue light and comparing theeffect on different images may be used to generate such segmentation, oreven a light of a frequency not visible to the human eye, such asinfrared, may be used.

Another method to find out the segments of image 620 that containintra-oral objects is by usage of image processing and analysistechniques. For example by filtering colors in order to locate teeth, orby finding the occlusal edge of the teeth, locating the separation linebetween intra- and extra-oral.

Lastly, a simpler segmentation may be performed using the orientation ofthe smart mirror. Particularly, by evaluating the angle between anhorizontal plane, parallel to the room's floor, and the normal to thesurface of the mirror at its center, and determining to turn on aupwards or downwards illumination based on a threshold. For example,above 20 degrees, select downward illumination, thus illuminatingobjects in the lower section of the mirror, and below 20 degrees selectupward illumination, illuminating objects in the upper section.Additionally, the selection of which light sources to be lit also dependon the rotation of the mirror around its center, as this valuedetermines which light sources are upward and which are downward.

Once the segment that requires illumination is determined, light sources107 that emit light in the direction of objects included in the segmentmay be turned on, and light sources 107 that emit light outside thesegment may be turned off, as illustrated in FIGS. 7A-C.

FIG. 7A illustrates a diagram 700 with viewfinder mirror 103 including aplurality of light sources 107A-B. Light sources 107A-B may emit lightat a first angle. In diagram 700, viewfinder mirror 103 also includes aplurality of light sources 707A-B, which emit light at a second angle,different from the first angle. In addition, the angles may be variable.Actuators (not shown) in viewfinder mirror 103 may enable respectivelight sources 107 and 707 to emit light in a variety of angles. Whetherfixed or variable, having different light sources at different anglesenables the device to select the direction of the light emitted. And byselecting the direction of the light emitted, the smart mirror devicemay direct the light to illuminate the object of interest and to avoidshining light toward the health care provider, dazzling her.

To illuminate an object of interest, various beams of light emitted fromdifferent light sources 107A-B or 707A-B may converge at the object. Indiagram 700, light source 707A and light source 107B are selected toilluminate an object 702. A beam of light 708A is emitted from lightsource 707A and a beam of light 708B is emitted from light source 107B.Both beams converge at object 702.

Similarly, FIG. 7B shows a diagram 730 that illustrates light source107A and light source 707B being selected to illuminate an object 732. Abeam of light 738A is emitted from light source 107A and a beam of light738B is emitted from light source 707B. Again, both beams converge atobject 732.

As described above, the object 702 and 732 may be determined varioustechniques, including face detection and image processing techniques. Inthis way, by selecting the direction of light to illuminate the portionof the image determined to contain the object of interest, embodimentcan illuminate the portion of the image sensor's field of view visibleto the health care provider, as illustrated in diagram 760 in FIG. 7C.

C. SYSTEMS AND METHODS UTILIZING THE INTRAORAL MIRROR

As discussed above, embodiments provide for a camera to be integratedonto an intraoral mirror and enable the mirror to act as a viewfinderfor the camera. Now, various systems and methods that utilize theintraoral mirror are described.

FIG. 8A illustrates a room 800 of a dental office including a system fordisplaying what is apparent to the healthcare provider in an intraoralmirror. Room 800 shows a healthcare provider 810 utilizing smart mirrordevice 100 to examine a patient 808. Tablet 806 shows what is visible tothe doctor in smart mirror device 100. Also inside room 800 is a basestation 804. Base station 804 includes cradles 812 and 814. Base station804 allows multiple smart mirrors (not shown) to be used in the sameroom 800. For example, an assistant may use one mirror on a patientwhile a dentist uses another on the same patient. Base station 804includes multiple cradles to allow multiple smart mirrors to dock.

Once healthcare provider 810 is no longer using smart mirror device 100,he may place it on one of cradles 812 and 814. When smart mirror device100 is docked with cradles 812 or 814, base station 804 may charge smartmirror device 100. Also, as described above, when docked with cradles812 or 814, smart mirror device 100 may calibrate its gyroscope andaccelerometer.

Base station 804 also provides for communication with smart mirrordevice 100. In particular, smart mirror device 100 transmits images andother information to base station 804, which transmits the informationfor display on tablet 806. The communication paths are illustrated inFIG. 8B.

FIG. 8B is an architecture diagram 850 of the system in FIG. 8A. Inaddition to the components in FIG. 8A, diagram 850 shows a foot pedal852, a medical records server 856, and a medical records database 858.

Smart mirror device 100, tablet 806, and foot pedal 852 may be connectedusing a wireless connection, such as Wi-Fi. In particular, base station804 may act as a Wi-Fi router and provide network routing and addressinformation to smart mirror device 100, tablet 806, and foot pedal 852.

Foot pedal 852 provides a way for the healthcare provider to inputinformation in a hands-free manner. For example, to request thecapturing of an image snapshot or a video clip. In one embodiment, smartmirror device 100 may include a microphone, and the healthcare providermay indicate when he would like to record information from themicrophone by depressing foot pedal 852.

Base station 804 is connected to medical records server 856 via one ormore networks 854, such as the Internet. Base station 804 may beconnected to the Internet either through a wireless or wired LAN in thedental office. Server 856 is a computerized process adapted to run inone or more remote server computers. Server 856 may, for example, be acloud server. Server 856 is further connected to an archival medicalrecords database 858. Medical records database 858 stores medical recordinformation, including dental status information collected from smartmirror device 100.

FIG. 9 is a block diagram 900 illustrating a possible configuration ofsmart mirror device 100. As discussed above, smart mirror device 100 hastwo separable components, oral piece 102 and hand piece 110, and oralpiece 102 includes viewfinder mirror 103, image sensor 204, lightsource(s) 107, and a connector 105.

It may be appreciated for those skilled in the art that a plurality ofsignal lines or buses 917 may exist, thus different components may belinked by different signal lines or buses 917, and that a signal linesor buses 917 depicted in the schematic diagram may represent a pluralityof such.

As discussed above for FIG. 1, viewfinder mirror 103 is a mirror, thushaving a reflective area. The reflection from viewfinder mirror 103provides visual guidance to a user about the objects that may beincluded in images captured by image sensor 204. As mentioned above,viewfinder mirror 103 may be round. In some embodiments, viewfindermirror 103 is planar. In some embodiments, viewfinder mirror 103 iscurved, concave, or convex. In some embodiments, viewfinder mirror 103has a spherical shape. In some embodiments, viewfinder mirror 103 has arectangular shape. It can be appreciated by those skilled in the artthat smart mirror device 100 can be embodied with different shapes ofviewfinder mirror 103 and/or a plurality of viewfinder mirror 103without departing from the spirit of this invention.

Viewfinder mirror 103 includes a pass-through 914. Pass-through 914allows the pass-through of light or visual information to allow light orvisual information to reach image sensor 204 (so that a respective imagecan be captured) or to allow illumination of objects by light from lightsource 107. In some embodiments, pass-through 914 is an opening inviewfinder mirror 103. In some embodiments, pass-through 914 is atransparent or semi- or partially-transparent area in viewfinder mirror103. In some embodiments, pass-through 914 includes an optical lens. Insome embodiments, pass-through 914 is a section of the area ofviewfinder mirror 103 that becomes transparent or partially transparentwhen light, possibly of an intensity above some threshold, is present.In some embodiments, pass-through 914 is a section of the area ofviewfinder mirror 103 that becomes transparent or partially transparentwhen electrical current or voltage is present. Pass-through 914 can belocated at the center of or at the perimeter, or at other locations of aviewfinder mirror 103. In some embodiments, pass-through 914 providesphysical separation from the environment to other of oral piece 102'scomponents, for example image sensor 204, when oral piece 102 isoperating in an intraoral environment, or during sterilization or thelike.

A plurality of pass-throughs 914 may exist. For example, smart mirrordevice 100 could include a rounded viewfinder mirror 103 having onepass-through 914 at its center (allowing visual information to reachimage sensor 204) and a plurality of pass-throughs 914 equidistant alongits perimeter (allowing light from a plurality of light sources 107 toprovide illumination).

Image sensor 204 captures still or video digital images. In someembodiments, image sensor 204 is an image sensor, or plurality thereof,that includes a pixel array, such as a charged coupled device (CCD), ora complementary metal-oxide-semiconductor (CMOS) sensor, or the like. Anexample of an image sensor is the MT9V023 available form ONSemiconductor of Phoenix, Ariz. In some embodiments, image sensor 204 ispart of a system-on-chip (SOC) with image sensing capabilities. The SOCthat may include a memory and/or an image signal processor (ISP) orother components. An example for such a SOC is the OV5640 available fromOmniVision Technologies Inc. of Santa Clara, Calif.

As described above, image sensor 204 is located relative to viewfindermirror 103 so that at least some of the objects reflected in viewfindermirror 103 are directly visible to image sensor 204 (so that they appearin a captured image). In some embodiments, image sensor 204 is locatedrelative to viewfinder mirror 103 so that at least some of the objectsreflected in viewfinder mirror 103 are indirectly visible to imagesensor 204. In some embodiments, image sensor 204 is located relative toviewfinder mirror 103 so that at least some of the reflective surface ofviewfinder mirror 103 is visible to image sensor 204. In someembodiments, image sensor 204 is located on or adjacent to thereflective area of viewfinder mirror 103. In some embodiments, imagesensor 204 is integrated into viewfinder mirror 103. In someembodiments, image sensor 204 is adjacent to pass-through 914. In someembodiments, a lens and/or a light pipe, such as an optical fiber,transmits visual information to image sensor 204. In some embodiments,image sensor 204 is physically separated from the environment (forexample an intraoral environment or sterilization environment) by apass-through 914.

Light source 107 illuminates objects in the proximity of smart mirrordevice 100. In some embodiments, light source 107 illuminates areas of aperson's mouth to improve the image reflected in viewfinder mirror 103or captured by image sensor 204. In some embodiments, a plurality oflight sources 107 are included. In some embodiments, light source 107emits light. In some embodiments, light source 107 transmits lightemitted elsewhere in smart mirror device 100. In some embodiments, theintensity of the light emitted or transmitted by light source 107 can becontrolled. In some embodiments, the intensity of illumination by aplurality of light sources 107 is concurrently controlled. In someembodiments, the intensity of each light source 107 of a plurality oflight sources 107 is independently controlled. In some embodiments, aplurality of light sources 107 all emit or transmit the same or similarlight wavelengths (or colors). In some embodiments, differentwavelengths (or colors) may be emitted or transmitted by a plurality oflight sources 107. In some embodiments, light source 107 is a ledemitting diode (LED). In some embodiments, light source 107 is a lightpipe, such as an optical fiber cable or the like. It can be appreciatedthat other devices can be used as light source 107 to illuminate areasof a mouth without departing from the spirit of the present invention.In some embodiments, light source 107 is a monochromatic light (alaser). In some embodiments, light source 107 transmits light emitted bya laser. In some embodiments, light source 107 is located at a perimeterof viewfinder mirror 103. In some embodiments, light source 107 may belocated at a different location of viewfinder mirror 103 and/orelsewhere in oral piece 102. In some embodiments, the light emittedand/or transmitted by light source 107 passes through a pass-through914. In some embodiments, light source 107 is physically separated fromthe environment (for example an intraoral environment or sterilizationenvironment) by a pass-through 914. In some embodiments, light source107 is located or directed towards the “back” of viewfinder mirror 103(further from the non-reflective area) of viewfinder mirror 103,providing ambient illumination as well as illuminating more of theenvironment space.

Connector 105 of oral piece 102 connects (physically and/orelectronically) to connector 922 of hand piece 110. Connector 922 may beconnector 113 illustrated in FIG. 1. Image sensor 204 and light source107 receive electrical power from hand piece 110 through connector 105.In addition, control and signaling is passed to and from image sensor204 and light source 107 through connector 105. For example, imagesensor 204 may transmit images captured via bus 917 to connector 105,which transmits images to hand piece 110. Similarly, connector 105 mayreceive and pass along control information indicating when and whetherto activate image sensor 204. For light source 107, connector 105 mayreceive commands on which light sources 107 to activate and when and howto activate them. Connector 105 is adapted to connect to a connector 922in hand piece 110. In some embodiments, connector 105 and connector 922may include a light connector, allowing transmission of light betweenhand piece 110 and oral piece 102.

Turning to hand piece 110, hand piece 110 includes an orientationmeasuring device 912, a user interface 924, a processor 923, a basestation connector 958, a connector 922, communication subsystem 929,power subsystem 921, and a memory 930.

Base station connector 958 enables hand piece 110 which may or may notbe attached to an oral piece 102 to dock with a base station. Thedocking may occur through a physical connection which holds hand piece110 at a predefined orientation. In addition, the docking may occurthrough a USB or near field communication connection or the like. Whendocking with the base station, hand piece 110 may receive electricalpower through base station connector 958, which may be used to chargepower subsystem 921. In addition, hand piece 110 may receive control andsignaling information through base station connector 958. For example,base station connector 958 may receive information needed to configure awireless communication connection between hand piece 110 and the basestation. Base station connector 958 may provide the wirelessconfiguration information (such as a service set identifier andpassword) to communication subsystem 929, as is discussed below. And,when docked to a base station, base station connector 958 may signalorientation measuring device 912 or software in memory 930 to calibrate.

Power subsystem 921 stores power for smart mirror device 100 andprovides power to the other components of smart mirror device 100. Powersubsystem 921 may include batteries, such as AAAA batteries, or acapacitor.

Orientation measuring device 912 measures an orientation (includingx,y,z, position and yaw, pitch, roll direction) of viewfinder mirror 103or generates data that enables to calculate an orientation of viewfindermirror 103. In some embodiments, orientation measuring device 912 is anaccelerometer. An example of an accelerometer is MMA8453Q available fromNXP Semiconductors N.V. of Eindhoven, Netherlands. In some embodiments,orientation measuring device 912 is a gyroscope. An example of agyroscope is FXAS21002C also available from NXP Semiconductors N.V.

User interface 924 includes an audio input 925, audio output 926, andinput/output controls 927. Audio input 925 captures audial information.In some embodiments, audio input 925 includes a microphone. In someembodiments, audio input 925 captures human voice, for example, toenable a healthcare provider to dictate observations for a patient'smedical record. Hand piece 110 includes an audio output 926, which emitssounds. In some embodiments, audio output 926 includes one or morespeakers. In some embodiments, audio output 926 includes headphone jacksand/or headphones.

Input/output controls 927 can include buttons, lights, knobs, capacitivesensors, actuators for haptic feedback or the like for a user to controland/or receive feedback relating to processes in smart mirror device100, for example, to initiate audio recording or image capturing, or setan intensity of illumination.

Communication subsystem 929 allows hand piece 110 to connect to one ormore remote computational devices, including, for example, to a basestation, or to a general purpose computational device such as personalcomputer, a smart phone, a tablet or similar, or a specializedcomputational device such as to another smart mirror device or remotespeakers or the like. In some embodiments, communication subsystem 929is adapted to connect to a wireless network, including, but not limitedto, WiFi and/or Bluetooth. In some embodiments, communication subsystem929 is adapted to attach to a wired network, including, but not limitedto, Ethernet, USB or thunderbolt.

Memory 930 may include random access memory (RAM) and may also includenonvolatile memory, such as read only memory (ROM) and/or flash memory.Memory 430 may be embodied as an independent memory component, and mayalso be embedded in another component, such as processor 923 and/orimage sensor 204, or may be embodied as a combination of independent aswell as embedded, and/or a plurality of memory components is present,the invention is not so limited. Memory 930 is adapted to includesoftware modules (a module is a set of instructions). In particular,memory 930 includes a streamer module 953, identification module 954,power monitor module 955, HTTP server module 956, illuminationcontroller module 950, image control module 951, and orientationcalculator module 968.

Processor 923 is adapted to run instructions stored in memory 930.Processor 923 may be a micro-controller unit (MCU), a digital signalprocessor (DSP) and/or an Image/Video Processing unit or the likecomponents that run instructions. An example of an MCU is MSP432P401xavailable from Texas Instruments Inc. of Dallas, Tex. An example of aDSP is C5000 available from Texas Instruments Inc. of Dallas, Tex. Anexample of an image/video processor is OMAP3525 available from TexasInstruments Inc. of Dallas, Tex. One or more processor 923 may bepresent. Processor 923 may be an independent component, it may also beembedded in another component, such as in image sensor 204, or anycombination thereof.

Illumination controller module 950 controls the operation of lightsource 107. In some embodiments, illumination module 950 sets theintensity of illumination of light source 107. In some embodiments,illumination module 950 receives a user request to increase or reduceillumination. In some embodiments, illumination module 950 receives userrequest to turn on or off some or all of light source 107. In someembodiments, illumination controller module 950 receives requests fromother software modules to increase and/or decrease illumination of oneor more of light source 107. In some embodiments, user input as well assaid requests are used to determine an intensity of illumination.

Orientation calculator module 968 reads data from orientation measuringdevice 912. Orientation calculator module 968 may for example integratedata from a gyroscope and accelerometer to determine a location (in, forexample, x,y,z coordinates) and a direction (for example, yaw, pitch,and roll). Because orientation calculator module 968 uses integration todetermine the location and direction of smart mirror device 100, errorsfrom the gyroscope and the accelerometer can accumulate over time.However, as described above, base station connector 958 may dock withthe base station in such a way to position hand piece 110 at a knownangle. When base station connector 958 is docked with the base station,base station connector 958 may signal orientation calculator module 968to calibrate. To calibrate, orientation calculator module 968 may setthe x, y, z, and yaw, pitch, and roll values to fixed values, such asthe value zero. Thus, when hand piece 110 is moved around, thecoordinate and direction values orientation calculator module 968determines may be relative to the coordinate and direction values set atthe base station.

Image control module 951 controls the capture of images and video, andaffects the output image quality. In some embodiments, image controlmodule 951 controls the intensity of illumination, for example, byrequests to illumination module 950, for example to improve theillumination conditions for a better image capture quality. In someembodiments, image control module 951 processes a set of time-successiveimages to create a single output image which has an improved visualquality, for example, but not limited to by selecting one image out ofthe set, or by combining portions of images, each portion from an imagein the set. In some embodiments, values indicating the acceleration ofimage sensor 204 when an image was captured are used to improve thequality of an output image, for example, but not limited to, selectingimages with least acceleration or interpolating among portions of two ormore images of different acceleration. In some embodiments, imagecontrol module 951 controls the aperture and/or focal point of a lens.In some embodiments, image control module 951 triggers the capture of asequence of images each with a different illumination. In someembodiments, image control module 951 triggers the capture of a sequenceof images each with a possibly different group of one or more of lightsources 107 set to illuminate, while the other one or more of lightsource 107 set to not illuminate. In some embodiments, image controlmodule 951 rotates an image, for example based on a rotation valuegenerated by orientation calculator module 968.

Identification module 954 identifies smart mirror device 100 to a remotecomputational device. In some embodiments, identification module 954implements an authentication handshake protocol in which theidentification occurs over a network session. In some embodiments,identification module 954 couples an identification to data prior to thedata being transferred to a remote computational device. Theidentification may include a globally unique ID for hand piece 110. Itmay also be timestamped and digitally signed.

Power monitor module 955 monitors the amount of energy available by thepower subsystem, and the power usage of smart mirror device 100. In someembodiments, power monitor module 955 receives a motion indicationgenerated by orientation measuring device 912, for example, but notlimited to, an acceleration indication. In some embodiments, powermonitor module 955 sets smart mirror device 100 into a standby mode whensmart mirror device 100 is not being used for a time interval largerthan some threshold. To set a standby mode, in which smart mirror device100 consumes a reduced amount of power, power monitor module 955 mayreduce or completely shut down the power supply to some of smart mirrordevice 100 components and/or alter or completely pause some of smartmirror device 100 software processes, or the like. In some embodiments,power monitor module 955 exits a standby mode, for example, by resumingpower to some of smart mirror device 100's components or resumingexecution of some of smart mirror device 100's processes, when anindication of usage of smart mirror device 100 is present. In someembodiments, power monitor 955 enters or exits a standby mode based onother parameters or indications, the invention is not so limited. Insome embodiments, power monitor 955 performs a shutdown, shutting powerto more (when compared to standby mode) or even all of smart mirrordevice 100 components, when an indication of not being used is presentfor a time interval larger than some threshold, or based on otherparameters or indications.

Streamer module 953 prepares and/or streams data to a remotecomputational device. The data can include video collected from imagesensor 204, smart mirror device 100's orientation and location collectedfrom orientation calculator module 968, audio input collected from audioinput 925, any data collected from input/output controls 927, powerrelated data collected from power monitor 955 and the specification ofhow light source 107 is illuminated from illumination controller module950. Streamer module 953 may associate data collected from these varioussources with each other. To associate data collected from differentsources, streamer module 953 may attach a timestamp. For example, eachframe in video image sensor 204 may include a timestamp which iscollected. Similarly, the orientation, audio, power, and input controlinformation may have a timestamp indicating when that information wascollected, and the illumination information may have a timestampindicating when light source 107 was illuminated in the mannerspecified.

In some embodiments, streamer module 953 formats images, video, audioand other data in a format for streaming to an application executing ona remote computational device via communication subsystem 929. In someembodiments, streamer module 953 formats images, video, audio and otherdata in a format suitable for streaming to an Internet browser, forexample, but not limited to, HTTP streaming, HTML, HTML5, RTSP, WebRTC.In some embodiments, streamer module 953 formats images, video, audioand other data with compression formats and/or format containers suchas, but not limited to, JPG, JPG 2000, MPEG-4, H.264, H.265, AAC, PCM,G.711, G.726, and the like. In some embodiments a proprietary format isused, the invention is not so limited.

In addition to streamer module 953, hand piece 110 may transmit datausing a HTTP server 956. In some embodiments, HTTP server 956 respondsto HTTP requests originating in a remote computational device.

Back to FIG. 7, in an embodiment, a viewfinder mirror of smart mirrordevice 100 includes light sources within the spectrum that enables thecuring of resin. A health care provider may illuminate the resin usingthe lights of a smart mirror device, while observing the progress of thecuring process in tablet 806. In addition to avoiding the exposure ofthe health care provider to a light that may be harmful to his eye, itenables to immediately continue the process of structure buildup,without requiring to switch instruments.

FIG. 10 is a flowchart illustrating an example method 1000 of the systemof FIG. 8A for distributed acquisition of images and possibly alsosupplementary data, and their storage for a subsequent processing, suchas the creating of a panoramic or immersive photograph, or thegeneration of digital dental impressions.

Method 1000 starts at step 1001 when a health care provider (HCP), suchas a dentist, a hygienist, or other, is performing an intraoralprocedure while, at least some of the time, the HCP observes thereflection of an intraoral environment area of interest in theviewfinder mirror of a smart mirror. This may occur in a variety of suchprocedures since an intraoral mirror is a frequently used instrument,including, but are not limited to, common procedures such as teethinspection, or a professional teeth cleaning. It can also be, but doesnot need to be, a dedicated image acquisition session, for example togenerate a digital impression.

At step 1002, the smart mirror captures images (still or video). In someembodiments time-tag supplementary data is optionally assigned to theimages. The time-tag indicates a time of image capture. In someembodiments, some or all of the smart mirror light sources are active,illuminating the intraoral environment when images are captured. In someembodiments, a light source illuminates with a light visible to thehuman eye, for example (but not limited to) “daylight”, which has acolor temperature of 5,500K-6000K, common in an electronic camera flash.In some embodiments, while capturing a plurality of images, the activelight sources vary, resulting in images of an object (or objects) in theintraoral environment, each captured with distinct illuminationconditions. Such plurality of images that differ in illuminationconditions may provide information useful for a 3D reconstruction of thescene being captured, for example, due to the different shadows present.In some embodiments, the variation of illumination conditions is suchthat the HCP may observe a plurality of illumination conditions. In someembodiments, the variation of illumination conditions is occurring at arapid pace, faster than may be perceived by the human eye, so thatpossibly to the HCP the illumination seems steady. In some embodiments,a light source illuminates with a light which is not within the humanvisible light spectrum, for example an infrared light (of wavelengthsaround 700-1600 nm), so that an HCP cannot perceive such illumination ora variation in such illumination, and yet allowing the capture of imagesby a sensor that is sensitive to such light.

At step 1003, a device identification supplementary data is assigned tothe images. The device identification identifies a particular smartmirror device. The device identification might be one or more serialnumbers, and/or an encoded message (for example an encrypted message),and/or an authentication key (for example a public key) or other.

At step 1004, orientation data, including position and/or directionsupplementary data, is assigned to the images. A position supplementarydata specifies, or can be used to calculate, a spatial location of apoint of interest on smart mirror, for example the position of thecenter of the viewfinder mirror or the position of the center of animage capture device, relative to a reference location. A directionsupplementary data specifies one or more angles pertaining to theangular position, representing the geometric rotations around a point ofinterest, for example the rotation around the center of the viewfindermirror or the rotation around the center of an image capture device,relative to a reference direction. In some embodiments, the positionand/or direction at the time of an image capture is assigned to animage. In some embodiments, the position and/or direction is tagged witha time-tag specifying the time at which said position and/or directionwere reached. In some embodiments, a plurality of positions and/ordirection along with their respective time-tags are assigned to one ormore images. It may be appreciated by those skilled in the art thatadditional embodiments may allow supplementing an image with informationregarding the position and/or direction reached at the time the imagewas captured, without departing from the spirit of this invention.

In some embodiments, assignment of supplementary data to an image isachieved by amending a digital representation of an image withsupplementary data, for example (but not limited to), by adding fieldsto an EXIF file (Exchangeable Image File Format) of an image. In someembodiments, assignment of supplementary data to an image is achieved bycoupling an image and coupling a supplementary data with correlatinginformation. For example, but not limited to, the digital representationof an image and the supplementary data are coupled with a patient'sidentification, and with correlating time-tags (e.g. relating to thesame time reference). It may be appreciated by those skilled in the artthat additional embodiments may allow assigning supplementary data to animage, without departing the spirit of this invention.

At step 1005, illumination conditions supplementary data is optionallyassigned to the images. Illumination conditions supplementary dataspecifies information regarding the light sources of smart mirror,including which of the light sources is active (and/or which is notactive), and possibly for each light source at what intensity, color orthe like attributes. In some embodiments, the illumination conditionsthat are present at the time of an image capture are assigned to theimage. In some embodiments, a set of time-tags with correspondingillumination conditions describe intervals at which the said conditionsremain unchanged, effectively describing temporal changes toillumination. In some embodiments, a plurality of illuminationconditions along with their respective time-tags are assigned to one ormore images. It may be appreciated by those skilled in the art thatadditional embodiments may allow supplementing an image with informationregarding the illumination conditions at which the image was captured,without departing from the spirit of this invention.

At step 1006, the images and supplementary data are transmitted to aserver. The transmission to the server may occur through the basestation.

At step 1007, images and other data received at the server are stored inan intraoral measurements database 1015 (a database organizes data sothat it is readily accessible).

FIG. 11 illustrates a block diagram 1100 further detailing components ofthe system of FIG. 8B. Like in FIG. 8B, diagram 1100 includes smartmirror device 100, tablet 806, medical records server 856, and footpedal 852, all in communication with base station 804. Foot pedal 852,smart mirror device 100, and tablet 806 may communicate with basestation 804 via a wireless interface 1136.

As detailed FIG. 9, smart mirror device 100 includes oral piece 102 andhand piece 110, which includes a streamer module 953 that streams datafrom smart mirror device 100 to base station 804. As shown in diagram1100, streamer module 953 includes a number of submodules, eachstreaming a different type of data to base station 804. In particular,streamer module 953 includes: a video streamer 1102 that streams videodata, audio streamer 1104 that streams audio data, an orientationstreamer 1106 that streams orientation and location data, anillumination streamer 1108 that streams illumination data, a powersteamer 1112 that streams power monitoring data, and a trigger streamer1110 that streams user input from smart mirror device 100, such asbutton presses. As discussed above, each of these data streams may becorrelated, for example with time-tags. Streamer module 953 streamsthese various data streams to base station 804.

Base station 804 includes an image analyzer module 1122, a face detectormodule 1128, a data streamer module 1120, a hand piece connector 1124,configurer module 1126, STT converter 1134, wireless interface 1136, anda dazzle avoidance module 1139. Image analyzer module 1122 analyzes theincoming video stream to capture what is visible to a healthcareprovider on smart mirror device 100's mirrored surface. To analyze theincoming video stream, image analyzer module 1122 includes an imagecropper module 1130, an image orientation module 1132, and asegmentation module 1138.

Face detector module 1128 receives an image from the image sensor anddetermines a location of the health care provider's face within thereceived image. In addition, face detector module 1128 may determine asize of the face within the received image. By determining a face'slocation and size, face detector module 1128 may determine a region ofthe image that includes the healthcare provider's face. Face detectormodule 1128 may be trained to locate a specific feature of thehealthcare provider's face, such as her eyes. Face detector module 1128may use any face detection algorithm. For example, face detector module1128 may use a genetic algorithm, a neural network trained to identifyfaces, or an eigen-face algorithm. In an embodiment, image analyzermodule 1122 uses data generated by face detector module 1128 togetherwith respective orientation data to determine a direction towards thehealthcare provider's face. The result is recorded as the latest knowndirection and may be used thereafter to estimate the position ofhealthcare's provider face in cases that they do not appear clearly inthe image.

Face detector module 1128 may detect a plurality of faces within thereceived image. In that case, face detector module 1128 may need todetermine which face corresponds to the healthcare provider. Facedetector module 1128 may determine the healthcare provider's face to bethe largest within the received image. Alternatively, face detectormodule 1128 may use the latest recorded direction towards the healthcareprovider's face as an input for its selection. Also, face detectormodule 1128 may recognize the healthcare provider's face using a facialrecognition algorithm. A facial recognition algorithm may operate bydetecting features of the face in the image and comparing with knownfeatures of the healthcare provider. In addition, the facial recognitionalgorithm could be used to identify and authenticate the healthcareprovider using smart mirror device 100.

Based on the location of the healthcare provider's face determined byface detector module 1128, image analyzer module 1122 determines aportion of the image visible to the health care provider on the mirrorsurface. Image analyzer module 1122 may apply the analysis, such ascropping, to additional images subsequent to the one detected by facedetector module 1128. Image analyzer module 1122 may operate asdescribed above for FIGS. 4, 6A and B.

In addition, image analyzer module 1122 corrects for distortions causedby the viewing angle of smart mirror device 100's lens. As mentionedabove, smart mirror device 100's lens may be a wide angle lens. Wideangle lenses can cause barrel distortion, where straight lines arecurved inwards in the shape of a barrel. Image analyzer module 1122 may,for example, correct for these barrel distortions.

Once image analyzer module 1122 corrects for any distortions anddetermines which portion of an image from a received video stream is ofinterest, image analyzer module 1122's image cropper module 1130 maycrop the determined portion from the image. In this way, image analyzermodule 1122 crops frames of the received video to more closely representwhat is apparent to the health care provider on smart mirror device100's reflective surface.

Image analyzer module 1122's image orientation module 1132 may correctfor the orientation of smart mirror device 100. Image orientation module1132 may correct for smart mirror device 100's orientation based on theorientation information received from smart mirror device 100 thatcorrelates to the image from the video. Using the orientationinformation, image orientation module 1132 may correct for smart mirrordevice 100's orientation as described above for FIGS. 5A-C and 6B.

Image analyzer module 1122 may conduct the above operations to crop andorient every frame of video streamed from smart device 100. After theimages are cropped and oriented, base station 804's data streamer module1120 may buffer in a memory and stream the information to tablet 806 fordisplay to the user. Additionally, base station 804's data streamermodule 1120 may buffer in a memory and stream images and otherinformation to medical records server 856 for analysis or archiving.Base station 804 also buffers full sized images together with location,illumination, and other data received from smart mirror device 100.

Image analyzer module 1122 may recognize an object in an image sensed bythe image sensor. For example, image analyzer module 1122 includes asegmentation module 1138 configured to determine an area of the imagethat shows the object inside a patient's mouth. Segmentation module 1138configured to segment the image into intra and extra-oral sections. Tosegment the image into intra and extra-oral sections, segmentationmodule 1138 may first receive a plurality of photographs taken by theimage sensor from substantially the same position with varying intensityof light emitted from the plurality of light sources. Then, segmentationmodule 1138 may compare brightness in a plurality of different portionsbetween the plurality of images. Finally, segmentation module 1138 maydetermine the intra-oral section as a portion in the plurality ofdifferent portions based on a variance in brightness between theplurality of images. For example, segmentation module 1138 that the agreater variance in brightness may be a surface close to the smartmirror device.

Dazzle avoidance module 1139 directs light emitted by the smart mirrordevice to avoid dazzling the health care provider. Dazzle avoidancemodule 1139 may direct light toward the intra-oral portion detected bysegmentation module 1138 or away from the health care provider's facedetected by face detector 1128. To direct light, dazzle avoidance module1139 may send commands to smart mirror device 100 to adjust thedirection or intensity of light emitted by its light sources, asdescribed above with respect to FIGS. 7A-C.

Tablet 806 executes a user interface application 1140, which is inwireless communication with base station 804. User interface application1104 is capable of displaying to a patient images collected from theimage sensor.

In addition to streaming video, a healthcare provider may be interestedin capturing an image at a particular point in time. To capture an imagein this way, a healthcare provider may select a button residing onhandpiece 110 of smart mirror device 100. When the healthcare providerselects the button, trigger streamer 1110 transmits to base station 804a trigger message with a time tag correlating to when the button wasselected.

When base station 804 receives the trigger message, image analyzermodule 1122 selects, from plurality of frames in the video, a pluralityof frames prior to the time when the button was pressed. When the buttonis pressed, the smart mirror device 100 is likely moved. By selectingframes immediately before the time when the button was pressed, theselected frames may more accurately represent the subject matter thatthe healthcare provider sought to capture. Once image analyzer module1122 selects a plurality of frames, image analyzer module 1122 blendsthe plurality of frames into a composite image. This process ofgenerating the composite image is described in greater detail withrespect to FIG. 16 below. The composite image may be presented to theuser and user interface application 1140 and may be archived in medicalrecords server 856.

In addition to the trigger button on smart mirror device 100, foot pedal852 may also be used for input, for example to trigger the capture of animage. In one embodiment, the button on smart mirror device 100 may beused to capture an image, while foot pedal 852 may be used to captureaudio. Foot pedal 852 is configured to wirelessly communicate with thebase station to provide user input to the base station. Foot pedal 852includes a trigger streamer module 1150.

When depressed, trigger streamer 1150 may transmit a message to basestation 804 indicating that foot pedal 852 is depressed and when footpedal 852 was depressed. Using this information, base station 804 cancorrelate with audio received from audio streamer module 1104 on smartmirror device 100.

Once the appropriate audio that was input while the foot pedal isdepressed is determined, speech to text converter 1134 can convert thedetermined audio into text. Speech to text converter 1134 may transmitthe audio to a cloud service that provides transcription. Thetranscription may then be stored in a database entry in medical recordsserver 856. Speech to text converter 1134 may recognize the text in astructured format. For example, the healthcare provider may recite thename of a tooth and, following the name of the tooth, recite itscondition. For example, the healthcare provider may state “rightmaxillary 2nd bicuspid” and state “cavity.” Recognizing this statement,speech to text converter 1134 may note a cavity in a database entry inmedical records server 856 corresponding to the right maxillary 2ndbicuspid.

In addition to providing audio input for conversion to text, audio inputof smart mirror device 100 enables to trigger actions by voice commands.Audio captured at the audio input is streamed to the base station, wherespeech to text converter 1134 analyzes it to locate certain keywords,for example the word “snapshot”, or “convert speech”. If such keyword isrecognized, the corresponding action takes place. For example, thesnapshot is taken or speech is converted into text for entry into thepatient's medical record.

Configurer module 1126 is configured to, when the dental mirror is inproximity to the base station, transmit, to smart mirror device 100,configuration information necessary to configure the wirelesscommunication between the dental mirror and the base station. In oneembodiment, configurer module 1126 is configured to transmit theconfiguration information when the dental mirror is docked with a handpiece connector 1124. In another embodiment, configurer module 1126 isconfigured to transmit the configuration information when the dentalmirror is in range for near field communication (NFC).

Configurer module 1126 operates to provide the necessary configurationinformation to tablet 806 and foot pedal 852 as well. When tablet 806 orfoot pedal 852 is in proximity to the base station, Configurer module1126 transmits to tablet 806 or foot pedal 852 configuration informationnecessary to configure the wireless communication between tablet 806 orfoot pedal 852. Configurer module 1126 may transmit the configurationinformation when it is in NFC range or when it is connected electricallyby a wire, such as a USB cable.

FIG. 12 is a flowchart illustrating a process 1200 to estimate theportion of the image captured by an image sensor in the intraoral mirrorthat is visible to a healthcare provider. In particular, process 1200calculates a translation adjustment to the center of interest in animage captured using a viewfinder mirror, according to some embodimentsof the invention. Process 1200 may, for example, be used in operation ofimage analyzer module 1122 in FIG. 11.

Process 1200 begins at step 1201 when an image is received. Then, atstep 1202 an estimation of the location (coordinates) of an observer'sposition within the frame of the image is calculated. In someembodiments, the coordinates of an observer's position are estimated byfinding the brightest area of the image and calculating a center in thatarea (assuming that in an image captured in an intraoral environment,using a viewfinder mirror, the brightest area indicates presence oflight arriving from the outside of a person's mouth). In someembodiments, a pattern of incisor (front tooth) is recognized and acenter of the gap between the positions of centers of upper and lowerincisors is used as an estimation of an observer's position. In someembodiments, a pattern of a part in the human face, such as an eye'siris or pupil is recognized, and the position of its center in the imageis used as an estimation of an observer's position. At step 1203, theshift or difference in image frame coordinates, between the center ofthe image and the estimated observer's position is calculated. At step1204 the calculated shift is used to calculate a translation adjustment.The shift and translation adjustment may be calculated as describedabove for FIG. 6A.

FIGS. 13 and 14 illustrate methods for rotating the image to adjust forthe orientation of the intraoral mirror. FIGS. 13 and 14 may be employedin operation of image orientation module 1132 in FIG. 11. FIG. 13 is aflow diagram illustrating a process 1300 for calculating an orientationof a viewfinder mirror, according to some embodiments of the invention.

Process 1300 begins at step 1301 when a calculation is triggered. Insome embodiments, the calculation is triggered by an orientationmeasuring device, for example when such device senses a change inorientation. In some embodiments, the calculation is triggered by atimer, for example, when at the end of a given time interval. In someembodiments, the calculation is triggered when the measure oforientation is requested by another process.

Once triggered, at step 1302, data from one or more orientationmeasuring devices is received. At step 1303, the data is processed. Insome embodiments, said data process includes combining said data withresults obtained in one or more preceding executing cycles of process1300. In some embodiments, said data process includes the integration ofsaid data over time. In some embodiments, said data process includes thecalculation of a moving window average over data. In some embodiments,said data process includes applying a signal processing filter, forexample a low pass filter. In some embodiments, no filtering isrequired, and said data is used as received. In some embodiments, thesaid data and/or the result of said data processing are adapted for useby one or more further run cycles of process 1300 and/or by anotherprocess. In some embodiments, other filtering techniques are used withthe purpose to smooth the output of process 1300. In some embodiments aplurality and/or combination of the above are used. In 1304 the result(or results) is stored for usage by another process or by a future runcycle of process 1300.

FIG. 14 is a flow diagram illustrating a process 1400 for calculating anadjustment to the orientation of a viewfinder mirror, according to someembodiments.

Process 1400 begins at step 1401, when an orientation calculation istriggered. Step 1401 is similar to step 1301 of FIG. 13. At step 1402,an image of teeth (intraoral or other) is received. At step 1403, imageprocessing is applied to the image (for example, but not limited toedge-detection and feature detection techniques) to find at least one ofthe following patterns:

(1) the line formed by the bottom edge of the top teeth (the contour ofthe incisal edges of the maxillary anterior teeth, sometimes called the“smile arc,” or the occlusal plane of the upper posterior teeth);

(2) the line formed by the top edge of the bottom teeth (the occlusalplane of lower incisors or posterior teeth), or

(3) the embrasures and/or visible lines between teeth (interproximalcontact areas, “connectors”), or gaps between teeth.

Each pattern is then represented as a vector.

At step 1404, the spatial geometry (defining which patterns are closerand which are further from the respective image capture device) of thepatterns found is estimated, by using at least one of:

(1) relative dimensions of patterns found;

(2) relative dimensions of objects;

(3) prior knowledge of patient's teeth and mouth structure;

(4) prior knowledge of standard dimensions.

At step 1405, a value is assigned to each pattern. The value serves as aweight of the respective pattern in the calculation of orientationadjustment. In some embodiments, the patterns found to be at a closerproximity receive a higher weight. In some embodiments, the patterns atthe center of the image receive a higher weight. In some embodiments,the patterns at the area in the image that is estimated to represent thecenter of viewfinder mirror receive a higher weight. In some embodimentssome patterns receive a weight of 1 and zero, one or more receive aweight of 0.

At step 1406, the spatial geometry generated in step 1404 and theweights generated in step 1405 are combined to calculate an estimateangle to use to adjusting the orientation of the image. The anglecalculated to position a tooth toward a top or bottom of the image suchthat the tooth has a portrait orientation.

At step 1407, the angle is processed as described above for step 1303and at step the resulting angle is stored at step 1408 as describedabove for step 1304.

FIG. 15A-B illustrate alternative methods to retrieve data from anintraoral mirror device. FIG. 15 is a flow diagram illustrating aprocess 1500 of user interaction with smart mirror, using an Internetbrowser executing in a computational device.

Process 1500 begins at step 1501 when a network published by smartmirror is selected. In some embodiments, the network is a WiFi network,and smart mirror broadcasts a Service Set Identifier (SSID) emulating aWiFi router or gateway or the like, so that a user may select it. Insome embodiments, a user selects a network previously known to thecomputational device. In some embodiments, a computational deviceautomatically selects smart mirror as a router. In some embodiments, auser otherwise chooses the network, the invention is not so limited. Insome embodiments, at step 1501 a smart mirror and a computational deviceare attached to a local area network (LAN) published by a router.

At step 1502, an Internet browser sends an HTTP request to smart mirror.In some embodiments, a user enters a Uniform Resource Locator (URL)and/or a Uniform Resource Identifier (URI) to the browser prior to thebrowser sending a request to smart mirror. In some embodiments, thebrowser sends a URL or URI or the like during its execution. In someembodiments, the browser sends a URL or URI or the like during theinitiation of its execution. In some embodiments, the browser sends anasynchronous message, for example an Asynchronous Javascript and XML(AJAX).

At step 1503, smart mirror receives an HTTP request, and responds with aweb page or other data. In some embodiments, data received by smartmirror may be processed and trigger certain actions, for example smartmirror may:

(1) receive a user and a password and enter a mode allowing additionaluser requests;

(2) receive a user and a password and store it in its internal memory;

(3) receive a selection of a local area network (LAN) and login data tosaid network;

(4) receive and store in its internal memory various configurationparameters;

(5) receive a request to stream images and/or audio and/or other, andthereafter perform one or more steps to respond to such requests, forexample, but not limited to, the steps described in process 1000 (FIG.10);

(6) receive and store in its internal memory a name to use as part ofits own identification. It might be appreciated that there are variousactions that an HTTP server implemented in a smart mirror can perform.

At step 1504, an Internet browser parses the data received from smartmirror and creates an appropriate presentation.

FIG. 15B depicts an embodiment of a system comprising a smart mirrordevice 1521 and an application 1522. Application 1522 is a set ofmodules (sets of instructions), stored or temporarily stored in a memoryof a remote computational device in implementing one or more of thefollowing computerized processes:

(1) search for one or more smart mirror devices;

(2) create a connection to a smart mirror device;

(3) receive a stream of video and/or audio and/or other data from smartmirror; or

(4) send a configuration parameter to a smart mirror device, and thelike processes. Such system provides an alternative to using an Internetbrowser (as described in process 1500, FIG. 15A), possibly providing animproved user experience.

FIG. 16 is a diagram 1600 illustrating how images from a stream ofimages in a video from the intraoral mirror are selected for capture.Diagram 1600 includes a chronological sequence of images 1602A-E. Images1602A-E may be frames from a video collected from a smart mirror device.In particular, images 1602A-E may be frames that have been processed,cropped and rotated as discussed above to reflect what is apparent tothe observer of the smart mirror device.

At a time 1604, a user of the smart mirror device makes an input torequest that an image be captured. Time 1604 occurs after image 1602D iscaptured but before frame 1602E is captured. The input may, for example,be a button press on the smart mirror device. Pressing a button maycause the smart mirror device to be moved away from the object that theuser intends to capture. This is illustrated, for example, in frame1602E. For this reason, frame 1602E is not selected for capture.Instead, at least one of the earlier images 1602A-D may be captured.

Moreover, several of the earlier images may be used to generate thecaptured image. The lighting, clarity, and focus may vary at each image.Thus, portions of the various images may be selected and blendedtogether. In diagram 1600, portion 1606 of image 1602A, portion 1608 ofimage 1602B, and portion 1610 of image 1602C are selected. The selectedportions are blended together to form a final composite image that iscaptured.

In this way, various systems and methods are presented that utilize inan intraoral mirror with an integrated camera.

D. UTILIZING AN INTRAORAL MIRROR TO GENERATE PATIENT STATUS

An intraoral mirror with an integrated camera can be utilized to capturea patient's dental status in a panoramic 2D photo (also called mosaic)and/or an immersive photo (also called 360 degrees photo, or sometimesvirtual reality photo) of a patient's mouth using images captured duringa routine dental treatment. The panoramic or immersive photographs maybe subsequently used by a dentist or other dental health care provider(HCP) to quickly become informed about a patient's status. Furthermore,a timed sequence of such photos that were created throughout years ofdental treatment, may be used as a history log of a patient's dentalstatus.

Such photo (or part thereof) created while a dental procedure is insession may be also used to identify the patient, in those cases were apanoramic and/or immersive photos (or parts thereof) of the patient fromprevious treatments exist. The photo (or part thereof) generated duringa treatment may be compared to those previously recorded, andsubsequently used for patient identification. If recognition of thepatient being treated is achieved, the system may thereafter assign anyphotos and/or other information collected during the treatment to thepatient, effectively selecting the patient's record.

An additional aspect of the invention is that the acquisition of imagesand other data used for the creation of panoramic or immersive photosmay possibly be distributed among a plurality of HCPs and/or a pluralityof sessions. As such, the invention described here allows for thegeneration of a panoramic and/or immersive photos to be performed inincremental steps.

The usage scenarios and embodiments are described to assist theunderstanding of the system and the methods that are later specified,and should not be viewed as limiting from other embodiments that do notdepart from the spirit of this invention.

FIG. 17A is a diagram 1700 illustrating how a feature from the interiorof the patient's mouth appears in different images taken from theintraoral mirror from different perspectives. In particular, diagram1700 shows a patient's mouth 1708 and depicts how a camera takes imagesfrom different perspectives, shown for example by perspective 1702 and1704. The image captured from perspective 1702 and 1704 are shown asphotographic images 1732 and 1734 of the interior of a patient's mouth.Images 1732 and 1734 are associated with position informationrepresented by perspective 1702 and 1704 respectively. Images 1732 and1734 are collected from an image sensor affixed to a dental mirror andwere taken such that at least a portion of images 1732 and 1734 overlapwith one another.

A feature detection algorithm may be applied to identify featuresappearing in the overlapping portions of both images 1732 and 1734. Anytype of feature detection algorithm may be used, includingscale-invariant feature transform (SIFT) or speeded up robust features(SURF). In this way, a common feature was identified between images 1732and 1734. The common feature appears at a point 1722 in image 1732 andat a point 1724 in image 1734.

Using points 1722 and 1724, a surface point may be determined. Inparticular, from the focal points of perspective 1702 and 1704, rays1712 and 1714 may be extended to traverse points 1722 and 1724. Rays1712 and 1714 may be extended according to a viewing angle at which theassociated images 1732 and 1734 taken. The intersection of rays 1712 and1714 is determined to be a surface point 1708.

FIG. 17B is a diagram 1730 illustrating how the points generated fromthe matching features are used to align different images to be stitchedinto a panorama indicating a patient's dental status, according to anexample embodiment. Diagram 1730 shows a plurality of images. Inparticular, diagram 1730 shows images 1732 and 1734 represented in FIG.17A. In addition, diagram 1730 shows images 1736, 1738, 1740, 1742, and1744.

Based on associated position information for the respective photographsand the matching features, images 1732, 1734, 1736, 1738, 1740, 1742,and 1744 are aligned to represent a series of adjacent teeth. Inparticular, images 1732-1744 may be adjusted such that the overlappingportions appear similar. The adjustment may be to account for variationsin scale between images 1732-1744 such that the resulting panoramicimage proportionally represents the patient's mouth. Images 1732-1744are also arranged on a two dimensional plane so that overlappingportions of images 1732-1744 appear at similar positions. For example,images 1732 and 1734 have an overlapping portion 1740 that shows anumber of features, including a feature 1748, a feature 1749, and thefeatures used to deduce surface point 1708 in the point cloud determinedas described above for FIG. 17A. Images 1732 and 1734 are scaled,positioned, and oriented such that feature 1748, feature 1749, andsurface point 1708 appear at the same location on a two-dimensionalgrid. In this way, images 1732 and 1734 are registered to locationswhere they are aligned.

Once images 1732-1744 are aligned, they can be stitched together into apanorama as illustrated in FIG. 17C.

FIG. 17C is a diagram 1750 illustrating stitching the images capturedfrom the image sensor into a panorama 1762 indicating the patient'sdental status. Stitching the images together in this way may involveblending images. To blend the images, they may be adjusted to accountfor variations in brightness between the plurality of photographs suchthat the panoramic image 1762 has a substantially uniform brightness. Inother words, the brightness is smoothed out across the image to avoid ablocky appearance and such that panoramic image 1762 appears to be asingle photograph. In addition to adjusting brightness, the stitchingmay involve adjusting hue of the image again to avoid a blockyappearance and to make panoramic image 1762 appear as a singlephotograph.

Such panorama images can be created for various surfaces of a patient'steeth. Front, top and back for the upper and lower teeth (facial/buccal,palatal/lingual, inscisal/occlusal).

Once the panorama image 1762 is created, different points on panoramicimage 1762 may be associated with different medical observations. Forexample, the history of each tooth may be recorded separately,associated with the position of the tooth and panoramic image 1762. Thismay involve associating a point on panoramic image 1762 to another imagerepresenting the same location in the patient's mouth. The other imagemay for example be an x-ray image.

FIG. 18A illustrates a diagram 1800 showing images captured at variousangles out of the same location inside the patient's mouth. Inparticular, diagram 1800 shows a patient's mouth 1808 and depicts how acamera takes images from the same location but at different angles,shown for example by perspective 1802 and 1804. The image captured fromperspective 1802 and 1804 are shown as photographic images 1832 and 1834of the interior of a patient's mouth. Images 1832 and 1834 areassociated with the angle represented by perspective 1802 and 1804respectively. Images 1832 and 1834 are collected from an image sensoraffixed to a dental mirror and were taken such that at least a portionof images 1832 and 1834 overlap with one another.

Based on associated position information for the respective photographsand the matching features, images 1832, 1834, 1836, and 1838 are mappedinto a sphere, each according to the angle which it was captured,relative to the center of the sphere which represents the image sensorlocation, as shown in FIG. 18B.

FIG. 18B shows a diagram 1840 with a sphere 1842 with pixels from images1832, 1834, 1836, and 1838 mapped to it. Using the sphere 1842, an imagecan be rendered to the user that provides a sense of depth, asillustrated in FIG. 18C.

FIG. 18C shows a diagram 1860 illustrating the geometry for rendering ofsuch an immersive photograph, where a virtual camera 1862 for thepurpose of rendering is placed at the center of the sphere. A healthcareprovider chooses an angle for the virtual camera and accordingly aviewport 1864 is determined. The pixels of the sphere are mapped intoviewport 1864 to produce an image for display. A healthcare provider maythem choose to “rotate” the camera around the center point, rotating theviewport as well for displaying various angles of the mouth around thesame center point.

Multiple such spheres are created, with center points in variouslocations inside the patient's mouth, allowing a healthcare provider tochoose the position of the center point, and hence the position withinthe mouth he is interested in, for locating the virtual camera.

When the virtual camera changes perspective, a different sphere 1804 isselected or a different portion of the sphere 1804 is mapped to viewport1802, resulting in rendering a different subject matter from theimmersive image.

E. UTILIZING AN INTRAORAL MIRROR TO GENERATE A DENTAL IMPRESSION

In additional to generating a dental status, embodiments generate highlyaccurate digital dental 3D impressions, even in the case where thesource for depth measurements are images. It generates the digitaldental 3D impressions without requiring a dentist (or other health careprovider) to dedicate an extended period of time for the intraoral dataacquisition. Instead of requiring a dedicated acquisition session, thedata needed to generate the impression is collected during a routinedental treatment. In addition, the data may be collected over the courseof a plurality of patient visits to one or more health care providers.

As shown in FIG. 17A, using points 1722 and 1724, a location of asurface point 1708 in an object captured by two or more images may bedetermined. In particular, measuring the distance between focal pointsof perspective 1702 and 1704, and extracting the angles of rays 1712 and1714 enables to calculate the measures of the triangle with edges 1722,1724 and 1708, hence the location of surface point 1708. A plurality ofsurface points located on objects inside the patient's mouth isthereafter used to create a three dimensional mesh model of the mouth.Other methods of determining the location of surface points of anobject, for example measuring the time-of-flight of radiation, such asLIDAR, or shape from shadow may also be used.

The amount of shadow casted by an object is generally related to thetopography of the object. When determining the shape of objects in themouth, the shadow that objects cast, as recorded in a two-dimensionalimage, may provide information about their three-dimensional shape. Tocollect data in order to generate 3-D impressions employing a techniquereferred to as shape-from-shadow, embodiments include varyingillumination image capturing as illustrated in FIGS. 19A and 19B.

FIGS. 19A and B are diagrams illustrating how varying illumination fromthe intraoral mirror can provide information on the three-dimensionalshape of a patient's teeth. FIG. 19A shows a diagram 1900 with a smartmirror device 1901. Smart mirror device 1901 has an image sensor 1906 atits center and a plurality of light sources at its perimeter. Theplurality of light sources include a light source 1902 and, opposite tolight source 1902, a light source 1904. Smart mirror device 1901 ispositioned to capture an image of a patient's teeth 1910, in particulara tooth 1909.

In diagram 1900, light source 1904 is illuminated, while light source1902 is not. Because of the varying illumination, tooth 1909 casts ashadow 1908. Image sensor 1906 captures an image, including both tooth1909 and shadow 1908. From the image, information about tooth 1909'sthree-dimensional shape can be determined. In particular, brightness ofthe image can be analyzed to determine where shadow 1908 is located. Thelocation of an image that occupies shadow 1908 is obscured by the 3Dshape of teeth 1910 from the perspective of light source 1904, while theremainder of the image is not.

FIG. 19B shows a diagram 1950 illustrating smart mirror device 1901illuminated differently from diagram 1900 in FIG. 19A. In diagrams 1900and 1950, smart mirror device 1901 has substantially the same positionand orientation with respect to teeth 1910. In diagram 1900, lightsource 1904 is illuminated while light source 1902 is not. But indiagram 1950, the opposite is true: light source 1902 is illuminatedwhile light source 1904 is not. The different illumination casts adifferent shadow 1952 on tooth 1909. Again, from shadow 1952, shapeinformation is determined. The location of an image that occupies shadow1952 is obscured by the 3D shape of teeth 1910 from the perspective oflight source 1902, while the remainder of the image is not.

Illumination between the various light sources may vary quickly, tooquickly for the human eye to detect. As the illumination varies, shapeinformation may be aggregated to generate a mesh of the surface pointsrepresenting the surface of a patient's mouth, and in particular ofteeth 1910. Additionally or alternatively, the light sources may emitlight outside the visible spectrum, such as infrared light. And theshadows detected may be shadows in this invisible light spectrum.

F. INCREMENTAL GENERATION OF STATUS AND IMPRESSION

Position information acquired in a smart mirror device and associated tocaptured images assists in the positioning of images, for example inorder to align them during a stitching process. The associated positioninformation is a value relative to a reference point, that is calibratedfrom time to time as described in FIG. 5A. Moreover, captured images ofthe same patient that were acquired by different healthcare providers indifferent dental office rooms, would exhibit position information valueseach relative to a different reference point. In order to use imagescaptured in various sessions, an alignment to the position of imagesmust be determined.

FIG. 20 illustrating a method for aligning the position informationacquired in one session to the position information acquired in anothersession. Thereafter the aligned information is used for generating apatient status or impression.

In some aspects of the invention, during a treatment, an HCP shifts thesmart mirror, for example, from a tray into the leftmost area in apatient's mouth, then about the oral cavity, and back to the tray. Thisexample is illustrated in FIG. 20 on a diagram illustrating a path 2003,an imaginary line tracking a route of a smart mirror 100 from a tray2005, into the mouth of a patient 808, around the oral cavity and out ofthe mouth. The relative position of the smart mirror is calculated, forexample, by integrating acceleration measurements over time. The resultis a set of position measurements that include a relative position andtheir corresponding time. The human oral cavity is within limiteddimensions. When a smart mirror is inserted to the oral cavity, it mustpass through the mouth, an opening that is within certain dimensions. Weshall call the elements of interest, an oral cavity, the mouth openingand the like as volume elements.

FIGS. 21A-B are flowcharts illustrating a method for aligning theposition information acquired in a session to the position informationacquired in a previous one or more sessions. They do so by analyzingpoint clouds of position measurements in order to identify and alignvolume elements, followed by a finer alignment using feature detectionin overlapping images. The aligned position information is then mergedinto one, incrementally generated point cloud model. Thereafter thepoint cloud model is used for generating an incrementally updatedpatient status or impression models.

At step 2101, the process is initiated. In some embodiments, theinitiation of the process 2100 is triggered when a server is idle, orwhen a server's resource utilization (computational power, memory, I/O,etc) becomes small enough, below a certain threshold. In someembodiments, the initiation of the process 2100 is triggered by a clock.In some embodiments, the initiation of the process 2100 is triggered bya user request. In some embodiments, the initiation of the photogeneration is otherwise triggered, the invention is not so limited.

At step 2102, a set of intraoral images and a set of supplementary datastored in an intraoral measurements database 2121 is selected for thepurpose of creating or updating a model. In some embodiments, the setcomprises some or all of the images and supplementary data that werecaptured in a single treatment session performed by an HCP, for example,as indicated in step 1007 of process 1000 (FIG. 10). In someembodiments, the set is comprised of some or even possibly all of theimages and/or some or even possibly all of the supplementary datapertaining to the same patient, and being captured in a plurality ofintraoral procedures, performed by one or more HCPs.

At step 2103, a point cloud is generated for the images and/orsupplementary data. A point cloud represents a collection of positionsand possibly also corresponding rotations. In some embodiments, a subsetof the set of supplementary data (that was selected at step 2102) isused to generate a point cloud. In some embodiments, each element in apoint cloud corresponds to a time on which an image was captured. Thepoint cloud may be acquired as described above for FIG. 20.

At step 2104, the point cloud generated in step 2103 (we shall call it:new point cloud), or part thereof, is aligned to a point cloud, or partthereof, generated in one or more previous sessions (we shall call it:previous point cloud) which is retrieved from a patient models database,see also the process of FIG. 21B. FIG. 21B details some of the sub-stepsof step 2104 of FIG. 21A. FIG. 21B details the analysis of the pointcloud in order to find volume elements and classify points (positions)into these volume elements. Once done, the volume elements can bealigned to the database model. For example, a volume element that isdetermined to be the opening of a mouth is found. Then, the mouthopening is aligned to one already found from previous sessions. In someembodiments, following an alignment, the new point cloud is merged to aprevious point cloud. By combining point clouds, the resulting combinedcloud may be less noisy and more accurate or may cover a greater portionof the patient's mouth.

At step 2105, images of objects of a spatial proximity are selected. Insome embodiments, one or more of said images are selected from the setof intraoral images (of step 2102). In some embodiments, one or more ofsaid images are selected from the set of intraoral measurements database2121 (hence may have been captured in one or more treatment sessions).In some embodiments, spatial proximity is determined by reference to themerged point cloud of step 2104. In some embodiments, spatial proximityis determined by matching features detected in a plurality of images,using image analysis methods. Matching features in images of spatialproximity are illustrated in FIG. 17A.

At step 2106, images are registered, so that image alignment is reached.When the images are captured, they may differ in scale and perspective.During registration, the images are adjusted to align the images so thatthe same object appears the substantially same in overlapping portionsof adjacent images. This may involve changing the image's scale orperspective, or correcting for any distortions. An example of howvarious images are aligned is depicted above for FIG. 17B.

At step 2107, irrelevant images are discarded. In some embodiments,irrelevant images include images that failed registration. In someembodiments, irrelevant images include images that do not show objectsof interest. In some embodiments, irrelevant images includes images oflesser quality. It may be appreciated by those skilled in the art thatthere are other reasons to classify images as irrelevant, the inventionis not so limited.

At step 2108, when updating status, an image is blended to a panoramicimage and/or to an immersive photo. In some embodiments, blendingincludes the smoothing of illumination (brightness) or hue amongadjacent images. An example of blending is illustrated above in FIG.17C.

Also, at step 2108, when updating an impression, depth information isextracted using images in spatial proximity, and merged into theimpression model. In some embodiments, images that have spatialproximity as well as a similar rotation are used to extract depthinformation. In some embodiments, said set of images vary in theillumination conditions present at the time of capture, so that suchimages show a variation in the shadows that intra-oral objects cast,effectively encoding the shape of the object, from the respective imagecapture pose (defined as a position and an orientation at time of imagecapture). Therefore a plurality of poses encode shape information from aplurality of directions, rendering a three dimensional model of anobject. An example of varying illumination for capturing images isillustrated above with respect to FIGS. 19A and 19B

At step 2109, the achieved models, and/or photos, or part thereof, arestored into a patient's models database 2120. In some embodiments, modelinformation includes the calculations and/or transformations to beapplied on a captured imaged to fit into a photo.

At step 2110, if the processing of images and/or supplementary data isexhausted, the process exits. Otherwise, it iteratively continues tostep 2105.

In some embodiments, additional conditions may cause process 2150 toexit or pause, for example, the server might become busy with othertasks of higher priority, or following the elapse of a time intervalallocated for the process and the like.

FIG. 21B is a flow diagram illustrating a process 2100 for classifyingpositions by volume elements. In some embodiments, such classificationis a step in aligning point clouds by aligning volume elements definedon the point clouds.

At step 2160, a set of positions and possibly direction (which may beincluded in a point cloud) are sorted according to time. The set iscomprised of measurements from a single session or part thereof, suchthat, during the time period, the patient's mouth and oral cavity are atabout the same position.

At step 2161, a set of vectors is defined. Each vector is based on twoor more temporally consecutive position measurements. Thereafter avector defines a line segment in space, and possibly also a direction(the direction would be, for example, an approximate direction ofmotion).

At step 2162, the spatial distances between vectors in a set arecalculated. In some instances, the spatial distance is the smallestpossible distance between any two points, each point on one of thevectors. In some embodiments, the spatial distance is the distance on aplane intersecting with the two vectors (when such intersection exists).In some embodiments, the spatial distance is the distance between themidpoints of the vectors. In some embodiments, the spatial distance isthe distance between the first or last points of the vectors. It may beappreciated by those skilled in the art that other possibilities todefine a distance between vectors may be used without departing from thespirit of the invention.

At step 2163, a plurality of sub-sets from the vector set are found,each sub-set includes all vectors in the set such that their distancesare smaller (and maybe also equal) than a higher limit, and possiblyalso larger (and maybe also equal) than a lower limit. The limit, forexample, may represents the largest opening of a human mouth, or, forexample, represents the largest diameter of a human oral cavity or, forexample, represents a small volume indicating that the instrument is atrest (i.e. on a dental tray), and the like. Thereby, each subset mayrepresent one or more volume elements. Each representation of a volumeelement by a sub-set shall be called a solution.

At step 2164, one or more of the solutions are rejected. In someembodiments, a solution is rejected if there exists a vector that is notin the respective sub-set but would have been included in the sub-set ifthe higher limit was larger, and/or the lower limit was smaller. In someembodiments, a solution is rejected if it is in proximity to anothersolution and/or subset. In some embodiments, a solution is rejected ifit is not in proximity to another solution and/or subset. In someembodiments, a subset is rejected based on image analysis, for example,a sub-set corresponds to a possible oral cavity, but the respectivecaptured images do not show intraoral visual characteristics (i.e.colors in some area, color histogram, dynamic range, objects etc.). Itmay be appreciated by those skilled in the art that other possibilitiesto reject a solution and/or a sub-set may be used without departing fromthe spirit of the invention.

At step 2165, the appropriate solutions are selected as a representationof volume elements, thereby defining volume elements on the point cloud.

Thereafter, at step 2166, points of the point cloud can be assigned tovolume elements, thus it can be determined for a point of the pointcloud if it is inside the volume element, for example the oral cavity,or it is not.

G. SYSTEM TO GENERATE A PATIENT STATUS AND DENTAL IMPRESSION

Generating the dental impression or generating the patient status mayrequire significant computing power. For example, depth measurementextraction and image registration may require significant memory orprocessing power. To deal with this, embodiments separate the impressionand status generation from the data acquisition. In particular, the datamay be acquired using a smart mirror device and base station in a dentaloffice, and the base station may transmit the data to a remote serverfor processing as illustrated in FIG. 22.

FIG. 22 illustrates a system 2200 with a server to stitch panoramas andimmersive photographs indicating a patient's dental status and togenerate dental impressions. Like the system in FIG. 8, system 2200includes smart mirror device 100, tablet 806, and base station 804,which is connected by one or more networks 854 to medical records server856. The records server 856 includes a panorama stitcher module 2212, apoint cloud generator module 2214, a dental impression generator module2216, and an immersive photograph stitcher module 2218. Medical recordsserver 856 is also coupled to a patient status database 2220 and patientmodel database 2222.

Medical records server 856 receives (i) a plurality of photographs ofthe interior of the patient's mouth, (ii) associated positioninformation for the respective photographs, and (iii) informationdescribing how the respective photographs were illuminated. Theplurality of photographs are collected from an image sensor affixed to adental mirror and overlap at least in part with another one of theplurality of photographs. Once collected, they are stored inmeasurements database 2220.

Point cloud generator module 2214 generates a point cloud from positioninformation and aligns it to previously generated point clouds, to allowthe alignment of images captured at different dental sessions. Pointcloud generator module 2214 may operate as described above with respectto FIG. 20, 21.

Panorama stitcher module 2212 determines features in the overlappingportions that match between multiple photographs. Based on associatedposition information for the respective photographs and the matchingfeatures, panorama stitcher module 2212 aligns the plurality ofphotographs to represent a series of adjacent teeth. Finally, panoramastitcher module 2212 stitches the plurality of photographs of theinterior of the patient's mouth into a panoramic image representing astate of at least a portion of the patient's mouth. Panorama stitchermodule 2212 may operate as described above with respect to FIGS. 17A-C.

Once generated, panorama stitcher module 2212 stores the panoramic imageinto patient model database 2222. Patient model database 2222 storeshistorical patient status information including historical panoramicimages of a patient's mouth. Patient model database 2222 may correlateportions of the panoramic images with the each other and with otherimages of a similar area of the patient's mouth. The other images may,for example, be x-ray images.

Once generated and stored, the panoramic images may be retrieved fordisplay. For example, the panoramic images may be displayed on tablet806, perhaps using a user interface application (not shown).

Immersive photograph stitcher module 2218 determines features in theoverlapping portions that match between multiple photographs captured atvarious angles out of the same center point position. Based onassociated position information for the respective photographs and thematching features, immersive photograph stitcher module 2218 maps theplurality of photographs onto the shape of a sphere. Then, Immersivephotograph stitcher module 2218 stitches the plurality of photographs ofthe interior of the patient's mouth into an immersive image representingthe captured images around the center point. Finally, Immersivephotograph stitcher module 2218 organizes multiple such spheresaccording to their center point, mapping positions in the patient'smouth to spheres. Immersive photograph stitcher module 2218 may operateas described above with respect to FIGS. 18A-C.

Once generated, Immersive photograph stitcher module 2218 stores theimmersive photograph into patient model database 2222. Patient modeldatabase 2222 stores historical patient model information includinghistorical immersive images of a patient's mouth. Patient model database2222 may correlate portions of the immersive images with the each otherand with other images of a similar area of the patient's mouth. Theother images may, for example, be x-ray images.

Once generated and stored, the immersive images may be retrieved fordisplay. For example, the immersive images may be displayed on tablet806, perhaps using a user interface application (not shown).

Not only can medical records server 856 generate a panorama indicatingpatient status, and immersive photographs, medical records server 856can also generate a dental impression. To generate a dental impression,medical records server 856 uses dental impression generator 2216 thatdetermines points on the surface of objects in the mouth, for example asshown in FIG. 17A and it may also use image analysis techniques,including the shape from shadow techniques, discussed above.

From the surface points, dental impression generator 2216 generates adental impression of the patient's mouth. The dental impression may be athree-dimensional mesh fit to the surface points.

In this way, using an intraoral mirror with an integrated camera, athree dimensional dental impression may be generated following a routinedental examination. The dental impression may be used, for example, forprosthodontics (such as making dentures, inlays and plastic casts),orthodontics, restorative dentistry (e.g. to make impressions of teethwhich have been prepared to receive indirect extracoronal restorationssuch as crowns or bridges), maxillofacial prosthetics (prostheticrehabilitation of intraoral and extra-oral defects due to trauma,congenital defects, and surgical resection of tumors) restorative,diagnosis and oral and maxillofacial surgery for both intra oral and orextra-oral aims (e.g. dental implants).

H. ILLUMINATING THE PATIENT'S MOUTH

As described above, the intraoral mirror may have a plurality of lightsources. How the light sources are controlled is described below withrespect to FIGS. 23 and 24.

FIG. 23 illustrates a method 2300 for setting illumination of anintraoral mirror with an integrated camera.

At step 2342, a set of light sources out of a plurality of light sourcecomponents are selected. At step 2343, each light source in the set isactivated (set to illuminate), and all other light sources aredeactivated (set to not illuminate). At step 2344, one or more imagesare captured. At step 2345, the one or more images are stored in memoryor transmitted to a remote computational device. At step 2346, there isa check whether more sets of source light are required, and if so thecapture process repeats with the new set.

The above process produces a set of images including (more or less) thesame objects but with various settings of illumination. As a result, the(more or less) same objects include different shadows and/or shading,which may allow production of an image with an improved visual quality,and/or reconstruct a 3-dimensional model of the object using so-calledshape from shadow (or shape from shading) techniques. Particularly, in aperson's intraoral environment, such sets of images captured with asmart mirror are characterized by the object of interest being in arelatively close proximity to the image capturing device and in anembodiment where a plurality of light source are located on theperimeter of a round mirror, due to such geometry, the shadows of suchobjects can be expected to be of enough difference to be conducive tosuch processes. In some embodiments, the intensity of illumination mayalso vary among the light source sets.

FIG. 24 illustrates a process 2400 for adjusting illuminationoriginating from the intraoral mirror.

At step 2482, whether the user can override the illumination intensityis determined. This may be set for example, in a user profile orconfiguration file. If a user overrides the illumination, process 2400proceeds to steps 2483 and 2495. At step 2483, the intensity value isset according to the user's request. At step 2495, one or more lightsources are set to illuminate and/or not illuminate respectively. Insome embodiments, a user sets a required intensity using a userinterface input/output control of smart mirror, for example a button orknob. In some embodiments, a user sets a required intensity using avoice request. In some embodiments, a user sets a required intensityusing a predefined configuration. In some embodiments, a user sets arequired intensity using a remote computational device.

If a user cannot override, process 2400 proceeds to step 2484, whereprocess 2400 checks whether another input overrides the illumination,for example an override can be generated by another software process(for example a process that requires control of illumination, such asillustrated in process 2300, FIG. 23). If so, process 2400 proceeds tosteps 2485 and 2495. At step 2485, the intensity value is set accordingto said request.

If neither a user nor other input overrides the illumination, process2400 proceeds to step 2486. At step 2486, the capturing and imageprocessing of one or more images is performed to establish a desiredillumination intensity for a one or more light sources. Such processingmay include, but is not limited to, identifying areas of an image thathave an intensity above or below a threshold, estimating from images orreceiving saturation and/or blooming data from an imaging sensor,comparing a plurality of images possibly with variable illumination andthe like.

At step 2487, the desired illumination is calculated. This may involveincreasing intensity when an image captured and processed at step 2486is determined to be too dark and decreasing intensity when an imagecaptured and processed at step 2486 is determined to be too light. Thedesired illumination may vary by light source. For example, if at step2486, a particular area of the image is determined to be darker than therest, a light source in proximity to the area may increase in intensity.Conversely, at step 2486, a particular area of the image is determinedto be lighter than the rest, a light source in proximity to the area maydecrease in intensity.

At step 2489, the calculated value is adjusted according to a userinput, if such input exists and is selected to affect the illumination,a condition checked at step 2488. At step 2491, the calculated value isadjusted according to another input, if such input exists and isselected to affect the illumination, a condition checked at step 2490.At step 2492, it is checked if the existing illumination is differentthan the optimal and if so it is adjusted (at step 2493). At step 2494,the result is filtered, so that changes in illumination are graduallyintroduced. At step 2495, the illumination intensity of one or morelight sources are set.

In this way, light sources on an intraoral mirror with an integratedcamera can be variably and automatically controlled to adjust forvariations in lighting.

I. MONITORING USAGE OF THE INTRAORAL MIRROR TO CONSERVE POWER

FIG. 25 illustrates a process 2500 for monitoring usage of the intraoralmirror to conserve power.

At step 2522, an “unused interval” timer, indicating a length of a timeinterval in which the smart mirror device is not being used, is reset.At step 2525, usage detection is validated. In some embodiments, usagedetection includes motion detection. In some embodiments, motiondetection is performed by processing measurements of acceleration ofsmart mirror in one or more axes. In some embodiments, motion detectionis performed by processing measurement of changes in angles of smartmirror relative to one or more axes. In some embodiments, motiondetection is performed by processing a set of temporally successiveimages captured by smart mirror, for example, identifying changes amongthe images. In may be appreciated that various ways to detect motion ofsmart mirror exist, the invention is not so limited. In someembodiments, usage detection includes analyzing the orientation of smartmirror, for example checking if it lays horizontally (if so it might beleft on table or a tray). In some embodiments, usage detection includesprocessing images captured by smart mirror, for example to assert thatviewfinder mirror is in an intraoral environment. In some embodiments, aplurality of such condition validations are used to determine whethersmart mirror is in use.

At step 2526, usage has not been detected, and an unused interval isdetermined. At step 2527, said interval is compared to a threshold. Atstep 2528, said interval surpasses a threshold, and smart mirror is setto a standby state in which its energy consumption is reduced, forexample, by reducing or cutting off the energy supply to one or more ofits components and/or suppressing or altering one or more of itssoftware processes.

At step 2529, usage detection is further validated. At step 2530, usagehas not been detected, and an unused interval is calculated, for aperiod in which smart mirror is in standby mode. At step 2531, saidinterval is compared to a threshold. If the interval surpasses athreshold, the smart mirror is shut down to further reduce its energyconsumption, for example, by reducing or cutting off the energy supplyto one or more or all of its components and/or suppressing or alteringone or more or all of its software processes. At step 2523, usage hasbeen detected and power mode is standby is checked.

At step 2524, smart mirror exits standby mode including, but not limitedto, resuming power to one or more of its components and/or resuming oneor more of its software processes.

J. ALTERNATIVE EMBODIMENTS OF THE INTRAORAL MIRROR

FIG. 26 is a diagram 2600 illustrating an alternative configuration ofthe intraoral mirror where an image sensor is placed on anon-autoclavable portion of the mirror.

In diagram 2600, hand piece 110 includes an insert 2604 adapted to slideinto a hollow part of appendage 104 of oral piece 102 through opening2606. In some embodiments, this configuration allows improved resilienceof oral piece 102 to a sterilization process, for example, by enablingto include electronic components in hand piece 110 instead of in oralpiece 102, hence sparing them the sterilization process. As anillustrative example, in some embodiments, hand piece 110 (orparticularly insert 2604) may include an image sensor 2602, instead ofthe sensor being included in oral piece 102. In this example, a prism,or light pipe or a small mirror internal to oral piece 102 (not shown)may transport light from semi-reflective surface 101 to image sensor2602.

In an intraoral mirror, it is common that a round mirror is attached atan angle to a handle. The presence of the angle allows a morecomfortable operation of an intraoral during treatment. However,viewfinder mirror 103 may be at a straight angle relative to appendage104 and tube 2604 may extend image sensor 2602 to semi-reflectivesurface 101, obviating the need for a light pipe. A joint may also beused to extend image sensor 2602 to semi-reflective surface 101,allowing mirror 103 to be angled relative to appendage 104, without theneed for a light pipe. This is illustrated in FIG. 27.

FIG. 27 is a diagram 2700 illustrating an alternative configuration ofthe intraoral mirror where an image sensor is placed on a jointedappendage of a non-autoclavable portion of the mirror.

In particular, in diagram 2700, oral piece 102 is configured such thatviewfinder mirror 103 is at an angle with (at least part of) tube 104.Insert 2706 of hand piece 110 has a joint 2704 connecting a fixedsection 2710 with a rotating section 2712. In some embodiments, arotating section 2712 can be at a varying angle relative to fixedsection 2710. In this way, as illustrated in FIGS. 26 and 27 an imagesensor may be placed on a non-autoclavable portion of a smart mirrordevice.

FIG. 28 is a diagram 2800 illustrating an alternative configuration ofthe intraoral mirror where the oral piece includes no electronics. Indiagram 2800, hand piece 110 extends the entire length of the smartmirror device. The portion of hand piece 110 that enters a patient'smouth is enclosed by two oral pieces: a front 2802 and a back 2804. Thefront 2802 and back 2804 attach to form an impermeable seal to coverhand piece 110 to ensure that hand piece 110 does not come in contactwith a patient's mouth. Hand piece 110 includes lights 107 and an imagesensor 2806, perhaps covered by a transparent window. Front 2802includes the mirror surface 2808, semi-reflective or transparent surface101 to allow light to pass through to image sensor 2806, and windows orlight pipes 2810 that allow light to be emitted from lights 107.

In another embodiment, the reflective mirror may be replaced by adisplay, and the image, perhaps processed as described above, from thesensor is displayed. Similarly, a semi-reflective mirror could conceal adisplay, so that, for example, a health care provider watches a tooth inthe reflective mirror and an x-ray is displayed to show, for example, aroot of a tooth.

In another embodiment, it may be desirable to have more hardware andprocessing in the oral, autoclavable portion of the smart mirror device,as illustrated in FIG. 29.

FIG. 29 is a block diagram illustrating an alternative embodiment of asmart mirror 2900. Smart mirror 2900 differs from smart mirror device100 described above principally in two respects. First, orientationmeasuring device 912 is located on oral piece 102, instead of hand piece110. Second, oral piece 102 includes additional processing capabilities.Oral piece 102 includes a processor 923 and a memory 930.

Processor 923 executes software stored on a memory 930. The softwareincludes an illumination controller module 950, image control module951, and orientation calculator module 968, as described above for FIG.1.

K. CONCLUSION

The databases disclosed herein may be any stored type of structuredmemory, including a persistent memory. In examples, this database may beimplemented as a relational database or file system.

Each of the processors and modules in FIGS. 9, 11, 22 and 29 may beimplemented in hardware, software, firmware, or any combination thereofimplemented on a computing device. A computing device can include, butis not limited to, a device having a processor and memory, including anon-transitory memory, for executing and storing instructions. Thememory may tangibly embody the data and program instructions. Softwaremay include one or more applications and an operating system. Hardwarecan include, but is not limited to, a processor, a memory, and agraphical user interface display. The computing device may also havemultiple processors and multiple shared or separate memory components.For example, the computing device may be a part of or the entirety of aclustered or distributed computing environment or server farm.

Identifiers, such as “(a),” “(b),” “(i),” “(ii),” etc., are sometimesused for different elements or steps. These identifiers are used forclarity and do not necessarily designate an order for the elements orsteps.

The present invention has been described above with the aid offunctional building blocks illustrating the implementation of specifiedfunctions and relationships thereof. The boundaries of these functionalbuilding blocks have been arbitrarily defined herein for the convenienceof the description. Alternate boundaries can be defined so long as thespecified functions and relationships thereof are appropriatelyperformed.

The foregoing description of the specific embodiments will so fullyreveal the general nature of the invention that others can, by applyingknowledge within the skill of the art, readily modify and/or adapt forvarious applications such specific embodiments, without undueexperimentation, without departing from the general concept of thepresent invention. Therefore, such adaptations and modifications areintended to be within the meaning and range of equivalents of thedisclosed embodiments, based on the teaching and guidance presentedherein. It is to be understood that the phraseology or terminologyherein is for the purpose of description and not of limitation, suchthat the terminology or phraseology of the present specification is tobe interpreted by the skilled artisan in light of the teachings andguidance.

The breadth and scope of the present invention should not be limited byany of the above-described exemplary embodiments, but should be definedonly in accordance with the following claims and their equivalents.

What is claimed is:
 1. A computer-implemented method for aligning three-dimensional dental data from a plurality of dental sessions, comprising: (a) receiving a plurality of point clouds, each point cloud: (i) representing three-dimensional locations in a dental room of a dental mirror including an interior of a patient's mouth, and (ii) collected during a different dental session when the patient's mouth is in a different position in the dental room; (b) aligning the plurality of point clouds to represent a common volume element within the patient's mouth; and (c) generating, based on the aligned plurality of point clouds, a three-dimensional model of the interior of a patient's mouth.
 2. The method of claim 1, further comprising: (d) evaluating whether respective locations in the plurality of point clouds are in the interior of the patient's mouth; and (e) rejecting locations not within the interior of the patient's mouth.
 3. The method of claim 2, wherein the evaluating (d) comprises, for respective point clouds in the plurality of point clouds: (i) determining a set of vectors such that each vector comprises a line segment between two or more locations in the respective point cloud, wherein the two or more locations were sensed consecutively and chronologically; (ii) determining spatial distances between vectors in the set of vectors; and (iii) based on the determined spatial distances, determining whether the associated locations in the respective point cloud are within the patient's mouth.
 4. The method of claim 3, wherein the determining (iii) comprises, for respective spatial distances: (i) determining whether the respective spatial distance is within a particular smaller and larger limit; and (ii) determining whether the locations associated with the respective spatial distance are within the patient's mouth based at least in part on whether the respective spatial distance is within the particular smaller and larger limit.
 5. The method of claim 2, wherein the three-dimensional locations are where an image sensor affixed to the dental mirror has captured a photographic image, wherein the evaluating (d) comprises, for respective three-dimensional locations, conducting image analysis on the photographic image captured at the three-dimensional location to determine whether the photographic image resembles the interior of the patient's mouth.
 6. The method of claim 5, wherein conducting image analysis comprises conducting histogram analysis.
 7. The method of claim 2, wherein the evaluating (d) comprises, for respective three-dimensional locations, determining whether the three-dimensional location is in proximity to a location rejected in (e).
 8. The method of claim 1, wherein the three-dimensional locations are where an image sensor affixed to the dental mirror has captured a photographic image.
 9. The method of claim 7, further comprising: (f) blending the photographic imagery data to account for variations in hue and brightness.
 10. The method of claim 1, further comprising: (d) determining that utilization of a computing device is below a threshold, and wherein steps (a)-(c) are executed on the computing device when the utilization is determined to be below the threshold.
 11. A non-transitory program storage device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform a method for aligning three-dimensional dental data from a plurality of dental sessions, comprising: (a) receiving a plurality of point clouds, each point cloud: (i) representing three-dimensional locations in a dental room of a dental mirror including an interior of a patient's mouth, and (ii) collected during a different dental session when the patient's mouth is in a different position in the dental room; (b) aligning the plurality of point clouds to represent a common volume element within the patient's mouth; and (c) generating, based on the aligned plurality of point clouds, a three-dimensional model of the interior of the patient's mouth.
 12. The program storage device of claim 11, wherein the method further comprises; (d) evaluating whether respective locations in the plurality of point clouds are in the interior of the patient's mouth; and (e) rejecting locations not within the interior of the patient's mouth.
 13. The program storage device of claim 12, wherein the evaluating (d) comprises, for respective point clouds in the in the plurality of point clouds: (i) determining a set of vectors such that each vector comprises a line segment between two or more locations in the respective point cloud, wherein the two or more locations were sensed consecutively and chronologically; (ii) determining spatial distances between vectors in the set of vectors; and (iii) based on the determined spatial distances, determining whether the associated locations in the respective point cloud are within the patient's mouth.
 14. The program storage device of claim 13, wherein the determining (iii) comprises, for respective spatial distances: (i) determining whether the respective spatial distance is within a particular smaller and larger limit; and (ii) determining whether the locations associated with the respective spatial distance are within the patient's mouth based at least in part on whether the respective spatial distance is within the particular smaller and larger limit.
 15. The program storage device of claim 12, wherein the three-dimensional locations are where an image sensor affixed to the dental mirror has captured a photographic image, wherein the evaluating (d) comprises, for respective three-dimensional locations, conducting image analysis on the photographic image captured at the three-dimensional location to determine whether the photographic image resembles the interior of the patient's mouth.
 16. The program storage device of claim 11, wherein the plurality of point clouds is generated from photographic imagery data and associated location information indicating a location where an image sensor captured the associated photographic imagery data.
 17. The program storage device of claim 16, wherein the image sensor is affixed to a dental mirror device.
 18. The program storage device of claim 17, wherein the associated location information is generated using a gyrocope calibrated when the dental mirror device is docked to a base station.
 19. The program storage device of claim 16, the method further comprising: (d) blending the photographic imagery data to account for variations in hue and brightness.
 20. The program storage device of claim 11, the method further comprising: (d) determining that utilization of a computing device is below a threshold, and wherein steps (a)-(c) are executed on the computing device when the utilization is determined to be below the threshold. 